CN108574701A - System and method for determining User Status - Google Patents
System and method for determining User Status Download PDFInfo
- Publication number
- CN108574701A CN108574701A CN201710133899.8A CN201710133899A CN108574701A CN 108574701 A CN108574701 A CN 108574701A CN 201710133899 A CN201710133899 A CN 201710133899A CN 108574701 A CN108574701 A CN 108574701A
- Authority
- CN
- China
- Prior art keywords
- data
- user
- biometric data
- biometric
- self
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 64
- 230000009471 action Effects 0.000 claims abstract description 57
- 230000004044 response Effects 0.000 claims abstract description 36
- 238000004891 communication Methods 0.000 claims description 17
- 230000029058 respiratory gaseous exchange Effects 0.000 claims description 16
- 239000008280 blood Substances 0.000 claims description 15
- 210000004369 blood Anatomy 0.000 claims description 15
- 230000007613 environmental effect Effects 0.000 claims description 11
- 230000036772 blood pressure Effects 0.000 claims description 8
- 230000008859 change Effects 0.000 claims description 8
- 230000036541 health Effects 0.000 claims description 7
- 239000000126 substance Substances 0.000 claims description 6
- 230000000007 visual effect Effects 0.000 claims description 6
- 230000004424 eye movement Effects 0.000 claims description 5
- 230000036387 respiratory rate Effects 0.000 claims description 5
- 210000005252 bulbus oculi Anatomy 0.000 claims description 4
- 239000000203 mixture Substances 0.000 claims description 4
- 239000004615 ingredient Substances 0.000 claims description 3
- 238000013500 data storage Methods 0.000 claims description 2
- 210000001097 facial muscle Anatomy 0.000 claims 2
- 230000008451 emotion Effects 0.000 description 53
- 230000000694 effects Effects 0.000 description 44
- 230000001360 synchronised effect Effects 0.000 description 43
- 238000003860 storage Methods 0.000 description 35
- 230000006870 function Effects 0.000 description 29
- 238000012544 monitoring process Methods 0.000 description 18
- 230000036651 mood Effects 0.000 description 18
- 241001269238 Data Species 0.000 description 17
- 238000004458 analytical method Methods 0.000 description 16
- 238000005516 engineering process Methods 0.000 description 13
- 238000004422 calculation algorithm Methods 0.000 description 12
- 230000033001 locomotion Effects 0.000 description 12
- 238000012545 processing Methods 0.000 description 10
- 230000008901 benefit Effects 0.000 description 9
- 238000005259 measurement Methods 0.000 description 9
- 238000012360 testing method Methods 0.000 description 9
- LFQSCWFLJHTTHZ-UHFFFAOYSA-N Ethanol Chemical compound CCO LFQSCWFLJHTTHZ-UHFFFAOYSA-N 0.000 description 8
- 238000005070 sampling Methods 0.000 description 8
- 206010016256 fatigue Diseases 0.000 description 7
- 230000006397 emotional response Effects 0.000 description 6
- 238000013213 extrapolation Methods 0.000 description 6
- 238000012552 review Methods 0.000 description 6
- 230000001351 cycling effect Effects 0.000 description 5
- 238000010586 diagram Methods 0.000 description 5
- 230000002159 abnormal effect Effects 0.000 description 4
- 230000005540 biological transmission Effects 0.000 description 4
- 235000013399 edible fruits Nutrition 0.000 description 4
- 230000002996 emotional effect Effects 0.000 description 4
- 230000008921 facial expression Effects 0.000 description 4
- 230000001965 increasing effect Effects 0.000 description 4
- 238000010801 machine learning Methods 0.000 description 4
- 238000003825 pressing Methods 0.000 description 4
- 238000012549 training Methods 0.000 description 4
- 102000009025 Endorphins Human genes 0.000 description 3
- 108010049140 Endorphins Proteins 0.000 description 3
- 241001465754 Metazoa Species 0.000 description 3
- 208000034972 Sudden Infant Death Diseases 0.000 description 3
- 206010042440 Sudden infant death syndrome Diseases 0.000 description 3
- 230000004913 activation Effects 0.000 description 3
- 238000013473 artificial intelligence Methods 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 3
- 230000002860 competitive effect Effects 0.000 description 3
- 229910003460 diamond Inorganic materials 0.000 description 3
- 239000010432 diamond Substances 0.000 description 3
- 239000003814 drug Substances 0.000 description 3
- 230000005055 memory storage Effects 0.000 description 3
- 238000005457 optimization Methods 0.000 description 3
- 238000011084 recovery Methods 0.000 description 3
- 210000002700 urine Anatomy 0.000 description 3
- 208000002193 Pain Diseases 0.000 description 2
- 206010041235 Snoring Diseases 0.000 description 2
- UCTWMZQNUQWSLP-UHFFFAOYSA-N adrenaline Chemical compound CNCC(O)C1=CC=C(O)C(O)=C1 UCTWMZQNUQWSLP-UHFFFAOYSA-N 0.000 description 2
- 238000013528 artificial neural network Methods 0.000 description 2
- QVGXLLKOCUKJST-UHFFFAOYSA-N atomic oxygen Chemical compound [O] QVGXLLKOCUKJST-UHFFFAOYSA-N 0.000 description 2
- 230000003190 augmentative effect Effects 0.000 description 2
- 244000309464 bull Species 0.000 description 2
- 238000012790 confirmation Methods 0.000 description 2
- 238000013135 deep learning Methods 0.000 description 2
- 238000003745 diagnosis Methods 0.000 description 2
- 230000002708 enhancing effect Effects 0.000 description 2
- 238000010195 expression analysis Methods 0.000 description 2
- 210000001508 eye Anatomy 0.000 description 2
- 230000001815 facial effect Effects 0.000 description 2
- 238000007689 inspection Methods 0.000 description 2
- 230000002452 interceptive effect Effects 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 238000010339 medical test Methods 0.000 description 2
- 230000004630 mental health Effects 0.000 description 2
- 238000003012 network analysis Methods 0.000 description 2
- 229910052760 oxygen Inorganic materials 0.000 description 2
- 239000001301 oxygen Substances 0.000 description 2
- 230000000704 physical effect Effects 0.000 description 2
- 230000035479 physiological effects, processes and functions Effects 0.000 description 2
- 238000007639 printing Methods 0.000 description 2
- 230000002829 reductive effect Effects 0.000 description 2
- 230000000717 retained effect Effects 0.000 description 2
- 238000012163 sequencing technique Methods 0.000 description 2
- 238000010025 steaming Methods 0.000 description 2
- 238000012546 transfer Methods 0.000 description 2
- 210000000707 wrist Anatomy 0.000 description 2
- 208000019901 Anxiety disease Diseases 0.000 description 1
- 241000208340 Araliaceae Species 0.000 description 1
- 241000238558 Eucarida Species 0.000 description 1
- 206010016275 Fear Diseases 0.000 description 1
- WQZGKKKJIJFFOK-GASJEMHNSA-N Glucose Natural products OC[C@H]1OC(O)[C@H](O)[C@@H](O)[C@@H]1O WQZGKKKJIJFFOK-GASJEMHNSA-N 0.000 description 1
- 206010019233 Headaches Diseases 0.000 description 1
- 206010020565 Hyperaemia Diseases 0.000 description 1
- 206010024264 Lethargy Diseases 0.000 description 1
- 206010068319 Oropharyngeal pain Diseases 0.000 description 1
- 235000005035 Panax pseudoginseng ssp. pseudoginseng Nutrition 0.000 description 1
- 235000003140 Panax quinquefolius Nutrition 0.000 description 1
- 201000007100 Pharyngitis Diseases 0.000 description 1
- 206010036590 Premature baby Diseases 0.000 description 1
- 206010037660 Pyrexia Diseases 0.000 description 1
- 206010044565 Tremor Diseases 0.000 description 1
- 230000001133 acceleration Effects 0.000 description 1
- 230000006978 adaptation Effects 0.000 description 1
- 230000036506 anxiety Effects 0.000 description 1
- 230000037147 athletic performance Effects 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 230000033228 biological regulation Effects 0.000 description 1
- 230000015572 biosynthetic process Effects 0.000 description 1
- 210000004556 brain Anatomy 0.000 description 1
- 230000007177 brain activity Effects 0.000 description 1
- 210000000481 breast Anatomy 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 238000005094 computer simulation Methods 0.000 description 1
- 239000012141 concentrate Substances 0.000 description 1
- 239000000470 constituent Substances 0.000 description 1
- 239000000599 controlled substance Substances 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 230000036461 convulsion Effects 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 230000001186 cumulative effect Effects 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000003066 decision tree Methods 0.000 description 1
- 238000000354 decomposition reaction Methods 0.000 description 1
- 230000007423 decrease Effects 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 239000007933 dermal patch Substances 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 230000010339 dilation Effects 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 208000002173 dizziness Diseases 0.000 description 1
- 230000035622 drinking Effects 0.000 description 1
- 230000005611 electricity Effects 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 230000005284 excitation Effects 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 210000000887 face Anatomy 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 230000005021 gait Effects 0.000 description 1
- 235000008434 ginseng Nutrition 0.000 description 1
- 239000011521 glass Substances 0.000 description 1
- 239000008103 glucose Substances 0.000 description 1
- 230000012010 growth Effects 0.000 description 1
- 230000037308 hair color Effects 0.000 description 1
- 231100000869 headache Toxicity 0.000 description 1
- 230000003054 hormonal effect Effects 0.000 description 1
- 229940088597 hormone Drugs 0.000 description 1
- 239000005556 hormone Substances 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 230000001771 impaired effect Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000004410 intraocular pressure Effects 0.000 description 1
- 238000011835 investigation Methods 0.000 description 1
- 230000001788 irregular Effects 0.000 description 1
- 150000002576 ketones Chemical class 0.000 description 1
- 238000002372 labelling Methods 0.000 description 1
- 238000007477 logistic regression Methods 0.000 description 1
- 210000004072 lung Anatomy 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 210000003205 muscle Anatomy 0.000 description 1
- 208000010125 myocardial infarction Diseases 0.000 description 1
- 230000006855 networking Effects 0.000 description 1
- 238000010238 partial least squares regression Methods 0.000 description 1
- 230000032696 parturition Effects 0.000 description 1
- 238000006116 polymerization reaction Methods 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 230000035485 pulse pressure Effects 0.000 description 1
- 210000001747 pupil Anatomy 0.000 description 1
- 238000012797 qualification Methods 0.000 description 1
- 238000007637 random forest analysis Methods 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 238000009877 rendering Methods 0.000 description 1
- 230000003252 repetitive effect Effects 0.000 description 1
- 230000000241 respiratory effect Effects 0.000 description 1
- 210000001525 retina Anatomy 0.000 description 1
- 230000033764 rhythmic process Effects 0.000 description 1
- 210000003296 saliva Anatomy 0.000 description 1
- 210000004761 scalp Anatomy 0.000 description 1
- 201000002859 sleep apnea Diseases 0.000 description 1
- 230000003997 social interaction Effects 0.000 description 1
- 238000012732 spatial analysis Methods 0.000 description 1
- 230000009295 sperm incapacitation Effects 0.000 description 1
- 238000013517 stratification Methods 0.000 description 1
- 238000012706 support-vector machine Methods 0.000 description 1
- 230000002459 sustained effect Effects 0.000 description 1
- 210000004243 sweat Anatomy 0.000 description 1
- 208000024891 symptom Diseases 0.000 description 1
- 238000003786 synthesis reaction Methods 0.000 description 1
- 230000009897 systematic effect Effects 0.000 description 1
- 230000002123 temporal effect Effects 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
- 230000036642 wellbeing Effects 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04M—TELEPHONIC COMMUNICATION
- H04M1/00—Substation equipment, e.g. for use by subscribers
- H04M1/72—Mobile telephones; Cordless telephones, i.e. devices for establishing wireless links to base stations without route selection
- H04M1/724—User interfaces specially adapted for cordless or mobile telephones
- H04M1/72448—User interfaces specially adapted for cordless or mobile telephones with means for adapting the functionality of the device according to specific conditions
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04M—TELEPHONIC COMMUNICATION
- H04M1/00—Substation equipment, e.g. for use by subscribers
- H04M1/72—Mobile telephones; Cordless telephones, i.e. devices for establishing wireless links to base stations without route selection
- H04M1/724—User interfaces specially adapted for cordless or mobile telephones
- H04M1/72403—User interfaces specially adapted for cordless or mobile telephones with means for local support of applications that increase the functionality
- H04M1/7243—User interfaces specially adapted for cordless or mobile telephones with means for local support of applications that increase the functionality with interactive means for internal management of messages
- H04M1/72439—User interfaces specially adapted for cordless or mobile telephones with means for local support of applications that increase the functionality with interactive means for internal management of messages for image or video messaging
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/442—Monitoring of processes or resources, e.g. detecting the failure of a recording device, monitoring the downstream bandwidth, the number of times a movie has been viewed, the storage space available from the internal hard disk
- H04N21/44213—Monitoring of end-user related data
- H04N21/44218—Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV program
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/47—End-user applications
- H04N21/482—End-user interface for program selection
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/85—Assembly of content; Generation of multimedia applications
- H04N21/854—Content authoring
- H04N21/8547—Content authoring involving timestamps for synchronizing content
Abstract
Provide a kind of method at least determining the state of user using self-report data and biometric data.This method includes：Receive the first biometric data of user；The first self-report data is received from user, wherein self-report data includes the state of user；And store the data in memory so that the first biometric data is in relationship and is linked to the first self-report data in time.Single file and/or at least one public id number can be used to be chained up data in relationship.Data can be in time chained up using at least one timestamp, offset, sample rate and/or hits.This method further includes receiving the second biometric data, the state for determining user of being come together with the data being stored in memory devices using it, and executes at least one action in response to this.
Description
Technical field
The present invention relates to the state for determining user, relate more specifically at least use readme (self-reporting) data
With biometric data the state of time and system and the side of at least one action are executed in response to this behind to determine user
Method.
Background technology
Recently, following equipment is had been developed for, can be surveyed with convenient form factor (form factor)
Amount, sensing or estimation and at least one or more relevant measurement (metric) (commonly known as biometric number of physiological characteristic
According to).For example, the equipment for having developed similar wrist-watch, the heart rate or pulse of individual can be measured, and using the data with
Other information age, the weight of individual (for example, etc.) result of calculation of coming together such as is burnt by individual given for one day
Total calorie.Similar equipment has been exploited for measuring, sense or estimating other kinds of measurement, name just a few for it is all
Such as blood pressure, breathing pattern, respiratory component, sleep pattern and blood alcohol content.These equipment are referred to as biometric device
Or biosensor metrology equipment.
Although the type sustainable growth of biometric device, keep relatively quiet using the mode of biometric data
State.For example, heart rate data is commonly used in providing the individual information of the calorie about its pulse and burning.By another example,
Blood alcohol and other data (for example, eye movement data) are commonly used in providing the individual letter about its blood alcohol content
Breath, and whether safely or legally can operate motor vehicles about them and inform individual.It is individual by another example
Breathing pattern (such as can measure by loudness level by decibel or by the variation of the decibel level in interval of time)
It can be monitored by doctor, nurse or Med Tech, to determine whether individual suffers from sleep apnea.
Although biometric data is useful at itself, if such data can be with other data (examples
Such as, video data etc.) combination, (for example, wirelessly, pass through network, etc.) be supplied to remote equipment, and/or can search for (example
Such as, allow certain conditions of such as raised heart rate etc promptly to be identified) and/or can intersection search (for example, using biology
Continuous data identifies the video section of diagram special characteristic, and vice versa), then it will more abundant information or dynamic.In this way
Data will can also indicate that individual specific time or in response to individual there are at least one things (for example, people, place,
Content of text (or the word being included therein or its theme), video content (or its theme), audio content (or are included therein
Word or its theme) etc.) in the case of how are you feeling today (for example, at least one affective state, mood, physical condition or spiritual shape
State).
Therefore, this will be advantageous, and there is demand to following system and method, be used alone or and its
He is used together institute at information (for example, at least one things, interesting data, at least one request (for example, problem, order etc.) etc.)
Determining state (for example, affective state, mood, physical condition or state of mind) is to generate some as a result, being such as supplied to a
Body some content (for example, some webpage, some advertisement etc.) and/or at least one action of execution based on web.Although to each
Known biometric state provides particular message understands with target alignment (target) to may not be to close for content originator
Reason, but the emotion of the mankind and mood provide the understandable specific background for being directed at message of content originator
(context)。
There is also the demand to following efficient system and method, can at least realize some or it is virtually all of
Aforementioned advantages, and can merge by usually using incompatible operating system each other or technology (for example, hardware platform, agreement,
Data type etc.) the data that are generated in the form of automatic or manual of various equipment.
In certain embodiments of the present invention, which is configured as in due course and cost-effective basis
The upper quantity for receiving, managing and filter information, and by accurately measuring, visualizing (for example, synchronously visual etc.), Yi Ji
The fast notification of the data point of definition or predefined range outer (or interior) can be with more valuable.
Such system and or method can with the biometric data (for example, heart rate etc.) measured while individual,
And/or the empirical data that " self-report data " that is collected simultaneously or examinee generate is together, by individual (for example, sportsman etc.) or he
The uses such as trainer, coach, with real-time (fact) or make later it is individual motion event (for example, jog, ride a bicycle,
Weight lifting is played soccer) performance during visualize, wherein individual during their practice, body-building or sports/training,
Or thinking to record and/or measure by user and/or user to its instant environment (that is, extraneous) factor, and/or physiology, feelings
During sense or the subjective response of physical condition are any other useful activity, themselves subjective body or spiritual shape are inputted
State is (for example, feeling the breaking-out of adrenaline " emerging in large numbers " or endorphin in systems, feeling fatigue, " cheering up again "
Deng), also, this with will usually medical treatment or the objective measurement system of laboratory environment be needed to be comparably relatively inconspicuous mode.
This will allow people (for example, trainer, third party etc. of individual, individual) such as to show above-mentioned in viewing or review (review) individual
In example in the performance of motion event or other body movements or in the case of health or affective state are just monitored or observe
While, the physiology and/or subjective psychology feature of monitoring/observation individual.This of self-report data can be realized by various methods
It is added timestamp in kind of input, including system automatically, on (enabled) body-building belt of smart phone, smartwatch, enabling
It speech explanation or annotation, short format or abbreviation key order or is utilized convenient for individual not interfere (or interfering as few as possible)
The input method of any other systematic connection of flow and practice in the afoot activity of individual.
Such system and or method is also by the remote reviewing in being applied convenient for such as tele-medicine and diagnosis, Chinese medicine
Business personnel or monitoring side or guardian need to the clear of patient or baby's identity and rapidly confirm and theirs visible
Physical condition, together with the biometric data and/or self-report data generated while them.
In addition, the system and or method should also be provided to examinee (subject) or monitoring side using video index
Mode, with for examinee the biometric history of oneself and/or compare for the data sample or population of other examinees
Data, effectively and intuitively to the data of examinee carry out benchmark test (benchmark), mapping and assessment, and with no matter which
A operating platform or application have been used to generate biometric and video information is unrelated.By that can filter/search for specific thing
Part (for example, biometric event, readme event, body event etc.), acquired data can be reduced or be edited (for example, wound
Build " excellent collection of choice specimens (highlight reel) " etc.), while keeping each video clip and data that are measured and/or collecting
Synchronization between (for example, biometric data, self-report data, GPS data etc.).This comprehensive index, the Yi Jiyong of event
It to execute the structure of related datas (video and other) with the data of (or not with) from other individuals or other relevant sources
The ability for changing polymerization can also be used to use the method for " big data " analysis and " machine learning " and add for being pushed away to action
The artificial intelligence (" AI ") for the implementation recommended and called provides more abundant horizontal information.
Invention content
The present invention provides a kind of exclusive use (in first part), processing, index, benchmark test, sorts, compares and show
Show biometric data or as a result, with other data (for example, video data, self-report data etc.) together (for example, same
Step) use, processing, index, benchmark test, sort, compare and show biometric data or the system and method for its result.This
The preferred embodiment of invention according to at least one external equipment (for example, the biometric for obtaining biometric data is set
Video equipment etc. standby, for obtaining video data) computing device (for example, smart phone etc.) of communication operates.In this hair
In bright first embodiment, the non-video data of the video data that may include audio data and such as biometric data is separated
Storage on the computing device and is linked to other data, this allow video and the search of non-video data with it is synchronous.
Of the invention (in second part) optimizes also directed to personalization preferences, or to individual/biometer from individual
The use for measuring data and/or self-report data, to determine at least one affective state, mood, physical condition or the spiritual shape of individual
State (" state "), then it is by individually or with other data (for example, at least close to individual when individual is undergoing emotion
One things, the interesting data etc. from the source of the data (for example, bid (bid) data etc.) based on web) be used together with
Certain data based on web are provided to individual or execute specific action.
May include more for executing using (for example, operation etc. on the computing device) about the first part of the present invention
Multiple modules of a function.For example, the application may include the video for receiving video data from internally and/or externally camera
Trapping module and for from internally and/or externally biometric device receive biometric data biometric capture mould
Block.Client platform can also include the subscriber interface module for allowing user to be interacted with platform, for editing regarding for video data
Frequency editor module disposes module, the database for replicate data and synchronization module, for handling for managing the file of data
The algoritic module of received data, the sharing module for sharing and/or storing data and be used for and third party social activity matchmaker
Body website (such as FacebookTM) connect (interface) center log in and ID modules.
These modules can be used for for example starting new session, receive the video data for session (that is, being caught via video
Obtain module), receive for session self-report data (for example, via keyboard etc.) and receive for session biometric number
According to (that is, via biometric trapping module).The data can be stored in local storage, in local data base, and/or
Remote memory storage devices (for example, in company's cloud or third party cloud service, such as DropboxTMDeng) on.In preferred embodiment
In, data be stored as so that its be linked to (i) identification session (for example, related data) it is synchronous with (ii) enabling (for example, when
Between (temporal) data) information.
For example, video data is preferably at least linked to time started (for example, at the beginning of session) and identifier.Mark
Symbol can be unique identification session individual digit or multiple numbers (for example, multiple globally unique identifier or general unique
Identifier (GUID/UUID)), the wherein work in the session of first digit unique identification and second digit unique identification session
It is dynamic, to allow session to include multiple activities.Identifier can also include session name and/or conversation description.About video counts
According to other information (for example, video length, video source etc.) (that is, " video metadata ") can also be stored and be linked to video
Data.Biometric and/or self-report data are preferably at least linked to the time started (for example, with the identical of video data is linked to
At the beginning of), identifier (for example, be linked to video data identical identifier) and sample rate, identification receives
And/or the rate of storage biometric and/or self-report data.
Once video, readme and/or biometric data are stored and are linked, so that it may to use algorithm to show data
Together.For example, if storing biometric data with the sample rates of 30 samples (spm) per minute, it can use and calculate
Method come video clipping (clip) start to show the first biometer magnitude (for example, below video data, it is folded with video data
Add), (two seconds in video clipping) the second biometer magnitudes of the display, (video after two seconds after two seconds
Four seconds in editing) display third biometer magnitude, etc..In alternative embodiments of the present invention, can with it is multiple when
Between stamp (for example, the respective timestamp for each storage value or offset (offset) or for each of each data type
From sample rate) storage non-video data (for example, biometric data, self-report data etc.), the timestamp can be with beginning
Time is used for together by non-video data and video data synchronization.
In one embodiment of the invention, biometric device may include the sensing for sensing biometric data
Device is used to interact (interface) with user and shows various information (for example, biometric data, setting data, operand
According to (start, stop and suspend) etc.) display, the memory for storing sensed biometric data, be used for
With the transceiver of exemplary computing device communication and for operating and/or driving transceiver, memory, sensor and display
The processor of device.Exemplary computing device includes the transceiver for receiving biometric data from exemplary biometric device
(1), for storing the memory of biometric data, for interacting and showing various information (for example, biometric number with user
According to, setting data, operation data (start, stop and suspend), the annotation in input session or add speech explanation etc.)
Display, is used for biometric data via because of spy the keyboard (or other users input) for receiving user input data
Net be supplied to main (host) computing device transceiver (2) and for operate and/or drive transceiver (1), transceiver (2),
The processor of keyboard, display and memory.
Keyboard (or other input equipments) in computing device or the alternatively keyboard in biometric device (or other
Input equipment) can be used for key in (enter) self-report data or about user the how are you feeling today of specific time data.
For example, if user feels fatigue, user can key in " T " on keyboard.If user feels that their endorphin acts as
With then user can key in " E " on keyboard.Also, if user cheers up again, user can key on keyboard
“S”.Alternatively, in order to further be convenient for operation during practice or sports, can pre-allocate such as " T ", " E " and
The short code key button of " S ", just as the speed dial phone number etc. of contact person frequently called, this can be on smart phone
It manually selects or is selected using speech recognition.Then the data (for example, entry or its expression) are stored and is linked to sample rate
(such as biometric data) or time stamp data can be the time for pressing each button or the offset with the time started.
This can allow self-report data synchronous with including video and other data of biometric data.It can also allow self-report data such as raw
Object continuous data is searched like that or filters ((when such as user starts to feel fatigue with particular event for example, in order to find
Deng) corresponding video).
In alternative embodiments of the present invention, computing device (for example, smart phone etc.) is also via wide area network (" WAN ")
(such as internet) is communicated with host computer device.The embodiment allows computing device to download application, unloading master from host computer device
At least some of above-mentioned function of computing device, and store data in host computer device (for example, allow individually or with
Such as biometric data video data synchronous with the non-video data of self-report data is watched by another networked devices).For example,
The software (for example, using, program etc.) operated on the computing device can allow user to play video and/or audio data, but
Be do not allow video and/or audio data are synchronous with biometric data.This may be because host computer device is used to deposit
The data (timestamp index, metadata, biometric data, sample rate etc.) to sync key are stored up, and/or in host computer device
The software of upper operation is necessary to synchronizing.By another example, the software operated on the computing device can allow user to broadcast
Put individually or the video and/or audio data synchronouss with biometric data, but may not allow computing device search or
Otherwise relevant portion (example is identified from biometric data extrapolation (extrapolate) or processing biometric data
Such as, can be used for creating " the excellent collection of choice specimens " of synchronous video/audio/biometric data) or to biometric and/or
Video data sequence (or the search of computing device can be limited or otherwise extrapolate or handle life from biometric data
Object continuous data come identify relevant portion (for example, its can be used for create synchronous video/audio/biometric data " essence
The color collection of choice specimens ") or the ability that sorts to biometric and/or video data).This may be because host computer device is used to deposit
The data (biometric data, biometric metadata etc.) to biometric data key of searching for and/or sort are stored up, and/or
Software is necessary to search (or executing Advanced Search) and/or sequence (or executing advanced sequence) biometric data.
In one embodiment of the invention, it can also include the video data of audio data in time " T " beginning and hold
The duration of continuous " n ".Video data is preferably stored in memory (locally and/or remotely) and is linked to other numbers
According to such as identifier, time started and duration.This data (for example, related data) at least bind video data
To specific session, it is specific at the beginning of, and identify including video duration.In the implementation of the present invention
In example, each session may include different activity.For example, taking a trip to (session) of Berlin can be related to cycling in the specific date
Across city (first activity) and walk through park (second activity).Therefore, identifier may include Session ID
Both with activity identifier, Session ID uniquely identifies session, activity identifier via globally unique identifier (GUID)
Activity is uniquely identified via globally unique identifier (GUID), wherein session/activity relationship is the relationship of father/son.
In one embodiment of the invention, biometric data stores in memory and is linked to identifier and sampling
Rate " m ".This allows biometric data linked in playback (for example, in relationship (relationally) and in time)
To video and/or self-report data.For example, if identifier is one, the time started is 1:00PM, video duration are one point
Clock, and 30spm when sample rate, then 2:00PM, which plays video council, to be caused 2:00PM will show the first biometer magnitude
(for example, below video, in the first-class of video), after two seconds will the second biometer magnitude of display (for example, below video,
In the first-class of video), and so on until video is 2:01PM terminates.Although self-report data can be as biometric data
It stores (for example, being linked to sample rate), still, if only periodically receiving this data, storage is linked to identifier
With the data of timestamp can advantageously, wherein " m " be receive self-report data time or the time with starting when
Between between offset (for example, 04 seconds etc. ten minutes after the time started).By the way that video and non-video data are divided each other
Storage is opened, can be easy searched for and synchrodata.
About by data link to the identifier that can be linked with other data (for example, time started, sample rate etc.), such as
The data are received when fruit, then the data can be linked to the identifier for current sessions (and/or activity).However, working as
When (for example, after conversation end) receives data after the fact, there are data can be linked to special session and/or work
The several ways of dynamic (or identifier associated with it).Data can link (for example, by user) or automatic via application manually
Link.About the latter, this can for example be accomplished by the following way：By the duration for the data that will be received (for example, video
Length) with session and/or compared with the movable duration；By assuming that the data and nearest session and/or activity phase that receive
It closes；Or by analyzing the data for including in the data received.For example, in one embodiment, including in the data of reception
Data (for example, metadata) can identify with the time of the data correlation and/or position, this can then be used to the number that will be received
According to being linked to session and/or activity.In another embodiment, computing device can show identification session and/or movable number
According to (for example, bar code, QR codes etc.).External video camera can be together with user and/or his/her ambient enviroment (example
Such as, before, later or period) record identification data (as shown in computing device).Then, using can be in video data
Middle search identifies data, and video data is linked to session and/or activity using the data.Then, applying when necessary can be with
Delete the identification division of video data.
About the second part that the present invention operates, web host can be via wide area network (WAN) and multiple content providers
(that is, source) and at least one network device communications, the wherein network equipment are by individual operations and are configured as the biology of individual
Continuous data passes to web host.Content provider provides content, such as website, webpage, image data, video to web host
Data, audio data, advertisement etc..Then, web host is configured as receiving biometric data from the network equipment, wherein biology
Continuous data is obtained from the individual of the operation network equipment or is associated with the individual of the operation network equipment.Then, using be used to from
The biometric data received determines at least one emotion, mood, physical condition or the state of mind.This uses known calculation
Correlation (correlation) between method, biometric data and various states and/or be stored in memory devices from
Data are stated to complete.
In one embodiment of the invention, content provider can indicate interest in in particular emotion state
Body provides the data based on web.In another embodiment of the present invention, content provider can indicate interest in in response to
Things (for example, a people, a place, the theme of content of text, the theme of video content, audio content theme etc.) and pass through
The individual for the particular emotion gone through or other interested parties (friend, employer, caregiver etc.) provide the number based on web
According to.Interest can be simple "Yes" or "No", or can be more complicated, interest, content owner such as 1-10 grades (scale)
It is ready to show that the amount of money of (impression) payment (CPM) or content owner are ready to click payment (CPC) every time every time
The amount of money.
Individually or combine the interesting data of other data (for example, randomness, consensus data etc.) can be by applying
It should be supplied to individual content-data (for example, advertisement etc.) using to determine.For example, if interesting data includes being directed to specific feelings
Sense or the different bids of emotion-things relationship, then the advertisement with best bid can be supplied to by application experienced the emotion
Individual.In other embodiments, consider other data when providing content to individual.In these embodiments, it is wanted in selection
At least to consider interesting data when being supplied to the content of individual.
In the method for the present invention, biometric data is from individual reception and for (individually or and self-report data
The corresponding emotion of individual, such as happy, angry, surprised, sad, detest or frightened are determined together).It is to be understood that emotion point
Class is stratification and this method can allow the more specific emotion of target alignment (such as wild with joy, amusement or comfort, this
A little is all the subset of happy emotion).About the emotion whether be individual current state or its whether based on individual to thing
Object (for example, a people, a place, to information etc. of individual display) response and make a determination.If the emotion is individual
Current state, then at least based on individual current affective state and interesting data select content.However, if the emotion is
Response of the individual to things, then being at least based on individual in the emotion response and interesting data selection of things (or its theme)
Hold.Then, selected content is supplied to individual or the network equipment by individual operations.
When executing specific action or executing specific request (for example, problem, order etc.), it is also contemplated that individual feelings
Sense, mood, body or the state of mind.In other words, before executing specific action (for example, the guidance in individual is inferior), network
Emotion, mood, body or the state of mind of individual can be considered in connection or network aware system or equipment.For example, being carried by individual
Divided together with the other biological continuous data that the order or instruction of confession can be related or from individual individually or with same individual
Analysis, to determine current emotional, emotion, body or the state of mind of individual.Then, network connection or network aware system or equipment
The state of individual can be considered when executing order or instruction.Depending on the state of individual, system or equipment is asked in execution
Action before can alert individual, or can additionally execute another action or replace it is requested action and hold
Another action of row.Such as, if it is determined that the driver of vehicle is angry or (intoxicated) that make drunk, then vehicle
Warning can be provided in the forward direction driver to start the engine, maximum speed can be limited, or driver can be prevented to operate vehicle
(for example, being switched to autonomous mode etc.).
By considering following detailed description of the preferred embodiment, will be provided to those skilled in the art at least using
Readme and biometric data determine the realization of the system and method and the additional advantage and purpose of the state of user
More complete understanding.Appended attached drawing will be referred to, will be briefly described as follows first.
Description of the drawings
Fig. 1 illustrates it is according to an embodiment of the invention for use, handle and show biometric data and
System for keeping biometric data synchronous with other data (for example, video data, audio datas etc.)；
Fig. 2A illustrates according to another embodiment of the present invention for using, handling and showing biometric data and use
In the system for keeping biometric data synchronous with other data (for example, video data, audio datas etc.)；
Fig. 2 B illustrate according to still another embodiment of the invention for use, handle and show biometric data and
System for keeping biometric data synchronous with other data (for example, video data, audio datas etc.)；
Fig. 3 illustrates the exemplary of the video data synchronous with biometric data according to an embodiment of the invention
Display；
Fig. 4 illustrate it is according to an embodiment of the invention for use, handle and show biometric data and
Block diagram for keeping biometric data synchronous with other data (for example, video data, audio datas etc.)；
Fig. 5 illustrates according to another embodiment of the present invention for using, handling and showing biometric data and use
In the block diagram for keeping biometric data synchronous with other data (for example, video data, audio datas etc.)；
Fig. 6 illustrate it is according to an embodiment of the invention make video data it is synchronous with biometric data, operation video
The method of data and search biometric data；
Fig. 7 illustrates the exemplary aobvious of the video data synchronous with biometric data according to another embodiment of the present invention
Show；
Fig. 8 illustrate be preferably linked to identifier (ID) (such as relation data), time started (T) and end time or
The exemplary video data of duration (n) (for example, time data)；
Fig. 9 illustrates the exemplary identifiers (ID) (for example, relation data) including Session ID and activity identifier；
Figure 10, which is illustrated, is preferably linked to identifier (ID) (for example, relation data), time started (T) and sample rate
(S) the example biometric data of (for example, time data)；
Figure 11, which is illustrated, is preferably linked to identifier (ID) (such as relation data) and time (m) (for example, time number
According to) exemplary readme (or " from realize ") data；
Figure 12 illustrate it is according to an embodiment of the invention how can using sampling biometric data point come
Extrapolation other biological stoichiometric number strong point；
Figure 13 illustrate according to another embodiment of the present invention how can using sampling biometric data point come
Extrapolation other biological stoichiometric number strong point；
Figure 14 is illustrated how can be using time started and relative data (for example, sample rate etc.) come will be biological
Continuous data and self-report data and/or the example of video data synchronization (for example, related in time)；
Figure 15 depicts exemplary " logining (sign the in) " screenshot capture of application, which allows user at least to capture
User carries out the video and biometric data of motion event (for example, cycling etc.), and together with biometric data
(or synchronous) display video data；
Figure 16 depicts exemplary " creating session " screenshot capture for the application described in Figure 15, and user is allowed to create new meeting
Words；
Figure 17 depicts exemplary " session name " screenshot capture for the application described in Figure 15, and user is allowed to key in session
Title；
Figure 18 depicts exemplary " conversation description " screenshot capture for the application described in Figure 15, and user is allowed to key in session
Description；
Figure 19 depicts exemplary " session start " screenshot capture for the application described in Figure 15, shows real-time reception
Video and biometric data；
Figure 20 depicts exemplary " looking back session " screenshot capture for the application described in Figure 15, and user is allowed to play back later
Session；
Figure 21 depicts exemplary " graphical display option " screenshot capture for the application described in Figure 15, and user is allowed to select
The data (for example, heart rate data etc.) to be shown together with video data；
Figure 22 depicts exemplary " looking back session " screenshot capture, wherein video data and life for the application described in Figure 15
(or synchronous) display together of object continuous data；
Figure 23 depicts exemplary " map " screenshot capture for the application described in Figure 15, shows and is shown on Google Maps
The GPS data shown；
Figure 24 depicts exemplary " summary " screenshot capture for the application described in Figure 15, shows the summary of session；
Figure 25 depicts exemplary " biometric search " screenshot capture for the application described in Figure 15, allows user in life
Particular organisms metering matters (for example, particular value, particular range etc.) are searched in object continuous data；
Figure 26 depicts exemplary " the first result " screenshot capture for the application described in Figure 15, shows shown in Figure 25
Biometric event the first result and corresponding video；
Figure 27 depicts exemplary " the second result " screenshot capture for the application described in Figure 15, shows shown in Figure 25
Biometric event the second result and corresponding video；
Figure 28 depicts exemplary " session search " screenshot capture for the application described in Figure 15, allows user to search for and meets
The session of certain standards；
Figure 29 depicts exemplary " list " screenshot capture for the application described in Figure 15, shows and is marked shown in Figure 28
Accurate result；
Figure 30 is illustrated via wide area network (WAN) and at least one content supplier and at least one network device communications
Web host, wherein biometric data that the web host is configured to respond to such as to receive from the network equipment (or and its
Relevant data) to the network equipment provide certain contents；
Figure 31 illustrates the one embodiment for the web host described in Figure 30；
Figure 32 provides the example chart that different biometric datas is linked to different emotions；
Figure 33 is provided the different moods in different response chains to chart, different things and different interest level
Example chart；
Figure 34 illustrates method according to an embodiment of the invention, is come using the biometric data from individual
Determine at least one emotion of individual, and individually or jointly other data select to be supplied to using at least one emotion
The content of individual；
Figure 35 provides example biometric sensing data string according to an embodiment of the invention；
Figure 36 provides exemplary emotion response data string according to an embodiment of the invention；
Figure 37 provides exemplary emotion item data string according to an embodiment of the invention；
Figure 38 provides exemplary item data string according to an embodiment of the invention；
Figure 39, which is illustrated, to be communicated with multiple remote equipments via wide area network (WAN) and is configured with biometric number
According to come determine individual at least one state and using at least one state come execute at least one action have network work(
The equipment of energy；
Figure 40 illustrates the one embodiment for the equipment with network function described in Figure 39；And
Figure 41 illustrates method according to an embodiment of the invention, is come using the biometric data from individual
It determines at least one state of individual and executes at least one action using at least one state.
Specific implementation mode
The present invention is provided (in the first portion) for individually or with other data (for example, video data, readme number
According to etc.) together (for example, synchronous) uses, processing, index, benchmark test, sort, compare and show that biometric data or its are tied
The system and method for fruit.Although should be appreciated that herein according to certain biometric datas (for example, heart rate, breathing pattern, blood
Alcohol content etc.) describe the present invention the part, but the invention is not restricted to this, and can with any biometric and/
Or physical data is used in combination, the data include but not limited to oxygen content, CO2Content, oxygen saturation, blood pressure, blood glucose, lung work(
Energy, intraocular pressure, body and environmental condition (temperature, humidity, illumination level, height above sea level and air pressure), speed (walking speed, running speed
Degree), the position advanced and distance, respiratory rate, heart rate variability (HRV), EKG data, horizontal, consumption and/or burning of perspiring
Calorie, ketone, waste discharge amount and/or level, hormonal readiness, blood constituent, salivary component, audible horizontal (for example, snoring
Deng), emotional levels and variation, skin electroresponse, E.E.G and/or activity or other neurology measurement, sleep pattern, physical traits
(for example, height, weight, eye color, hair color, iris data, fingerprint etc.) or response are (for example, changes in faces, iris
(or pupil) variation, voice (or tone) variation etc.) or any combination thereof or result.
As shown in Figure 1, biometric device 110 can be communicated with computing device 108 (for example, smart phone), calculating is set
Standby 108 communicate via wide area network (" WAN ") 100 (for example, internet) and at least one computing device (102,104,106) again.
Computing device can be using one or different operating system or platform different type (for example, PC, laptop computer,
Tablet computer, smart phone, smartwatch etc.).In one embodiment of the invention, biometric device 110 is configured
For the heart rate (for example, biometric data) of acquisition (for example, measurement, sensing, estimation etc.) individual.Then, biometric data
It is provided to computing device 108, computing device 108 includes video and/or audio logger (not shown).
In the first embodiment of the present invention, video and/or audio data are carried together with heart rate data via network 100
Supply host computer device 106.Because simultaneous video and/or audio data and heart rate data are provided to host computer device
106, thus the main application (not shown) operated in host computer device can be used for synchronizing video data, audio data and/or
Heart rate data, to, allow user's (for example, via user calculating equipment 102,104) while watching biometric data,
Viewing video data and/or listen to audio data (in real time or having time lingeringly).For example, as shown in figure 3, main application can be with
Usage time stamp 320, or using other sequence (sequencing) methods of metadata, to make video data 310 and biology
Continuous data 330 synchronizes (also, in other embodiments, making video data 310 synchronous with self-report data), to allow user to see
See the individual for example in specific time 340 (for example, by 76 seconds time starteds) (for example, in patient, infanette in hospital
Baby etc.) and specific time 340 (for example, by 76 seconds time starteds) and the associated biometric data of individual (
In other embodiment and self-report data).
It should be appreciated that main application can be additionally configured to execute other functions, for example, video data, audio data, from
It states and searches for specific activity in data, biometric data and/or metadata, and/or to video data, audio data, readme
Data and/or biometric data sequence.For example, main application can allow user to search for specific biometric event, for example,
Already exceed the heart rate of specific threshold or particular value, have descended to specific threshold or particular value heart rate below, in minimum
Between specific heart rate (or range) etc. in section.As another example, main application can in time sequencing, it is big according to biometric
Small (magnitude) (from it is up to minimum, from most as low as highest etc.), according to comment (review) (from preferably to it is worst, from most
Difference is to preferably etc.) or according to viewing (from up to minimum, most etc. from being minimized to), to video data, audio data, life
Object continuous data or multiple synchronous editings (for example, excellent collection of choice specimens) sequence.It should be further understood that the ranking of such as data,
The function of search and analysis is not limited to individual sessions of user, but can cross over the user any amount of individual sessions,
And multiple users session or multiple sessions and execute.All various information (video, biometric and other information) this
One purposes of a set is the enough data points that can be generated for big data analysis and machine learning, it is intended in generation AI
Infer and recommends.
As an example, machine learning algorithm can be used for searching for video data automatically, most noticeable content is found, with
Afterwards, most noticeable content will be by engagement (stitch) together into short " the excellent collection of choice specimens ".Multiple sport can be used
The grading (rating) for the level interested that video and user from the user are in progress with video, to train neural network.
The input node of network can be the intermediate value of the excited grading of the sample and present frame of the image pixel intensities variation between frame.Machine
Device learning algorithm can also be used in combination with multilayer convolutional neural networks, to classify video content (for example, being in video automatically
What sports).Once content is either automatically or manually identified, so that it may to use algorithm by the activity of user and ideal
The activity of change is compared.For example, system can be by the videograph of the golf of user and professional golfers
Videograph is compared.Then, system can provide a user how can to improve their swing about user progressive
(incremental) prompt.The body-building level of user can also be predicted (for example, if they adhere to them using algorithm
Plan, then give them continue take exercise excitation), by user with have similar body-building level other users or practitioner
Matching, and/or it is created as the custom (routine) of each user optimization.
It is also understood that as shown in Figure 2 A, biometric data can be supplied directly to host computer device 106, and without
Cross computing device 108.For example, computing device 108 and biometric device 110 can be directly or via networks 100, independently
It is communicated with host computer device.It should be further understood that video data, audio data, readme and/or biometric data are not required to
It to be supplied to host computer device 106 in real time.For example, video data can be provided in later time, as long as data can be by
Identification is tied to special session.If video data can be identified, video data can be synchronized in real time
The other data (for example, biometric data, self-report data etc.) received.
In one embodiment of the invention, as shown in Figure 2 B, system includes computing device 200 (for example, smart phone),
Computing device 200 is communicated via WAN (see, for example, in Fig. 1 100) with the multiple equipment including host computer device 240, via WAN
(see, for example, 100 in Fig. 1) are communicated with third party device 250, and communicated with local device 230 (for example, via wireless or
Wired connection).In a preferred embodiment, computing device 200 from host computer device 240 (for example, company's cloud) download program or
Using (that is, client platform).Client platform includes being configured as executing multiple modules of multiple functions.
For example, client platform may include the video capture for receiving video data from internally and/or externally camera
Module 210 and for from internally and/or externally biometric device receive biometric data biometric trapping module
212.Client platform can also include the subscriber interface module for allowing user to interact (for example, providing self-report data) with platform
202, for editing the video editing module 204 of video data, for managing (for example, storage, link etc.) data (for example, regarding
Frequency evidence, biometric data, identification data, time started data, duration data, sampling rate, self-report data, ring
Border data, time stamp data etc.) file disposition module 206, for replicate data (for example, computing device 200 will be stored in
Data copy to host computer device 240, and/or copy the user data being stored in host computer device 240 to calculating and set
Standby database and synchronization module 214 200), for handle received data (for example, synchrodata, search/mistake filter data,
Create excellent collection of choice specimens etc.) algoritic module 216, for shared and/or store and be related to the data (example of individual session or multiple sessions
Such as video data, the excellent collection of choice specimens) sharing module 220 and for third party's social media website (for example,
FacebookTM) connect center log in and ID modules 218.Sharing module 220 can be used for for example via network (for example, because of spy
Net) at least one image of another equipment (for example, mobile device) offer and/or sound, wherein image and/or voice recognition is used
The state (or based on the state of user to select) at family.Such embodiment will allow such as user's directly or indirectly (example
Such as, by identifying or describing emoticon, color, image and/or the sound of their state) by their state (for example, working as
Preceding state) pass to friend, household etc..
Can be the computing device 200 of smart phone, tablet computer or any other computing device about Fig. 2 B, it can
To be configured as downloading client platform from host computer device 240.Once client platform is run on computing device 200, put down
Platform may be used for starting new session, be session reception video data (that is, via video trapping module 210) and be connect for session
Receive biometric data (that is, via biometric trapping module 212).The data can be stored in local storage, originally
In ground database, and/or remote memory storage devices are (for example, in company's cloud or third party cloud (such as DropboxTMDeng) in) on.
In a preferred embodiment, data are stored to the letter that data are linked to (i) identification session and (ii) enables synchronization
Breath.
For example, video data is preferably at least linked to time started (for example, at the beginning of session) and identifier.
Identifier can be the individual digit or multiple digital (for example, the multiple overall situation (or general) unique marks for uniquely identifying session
Accord with (GUID/UUID)), the first digit in multiple numbers uniquely identifies session, and the second digit in multiple numbers is only
One ground identifies the activity in session, to allow session (for example, going to travelling or the stroke of destination (such as Berlin)) to include
Multiple activities (for example, cycling, walking etc.).It is only used as example, movable (or session) identifier can have height
128 bit identifiers of probability uniqueness, such as 8bf25512-f17a-4e9e-b49a-7c3f59ec1e85.Identifier can be with
Including session name and/or conversation description.About video data other information (for example, video length, video source etc.) (that is,
" video metadata ") video data can also be stored and linked to.When biometric data is preferably at least linked to beginning
Between (for example, be linked to video data it is identical at the beginning of), identifier is (for example, with the identical of video data is linked to
Identifier) and sample rate, sample rate identify the rate that the biometric data is received and/or stores.For example, heart rate data
It can be with 30 samples (30spm) per minute (that is, every two seconds primary) or the rate of some other predetermined time interval sampling
It is received and stored.
In some cases, the sample rate that platform uses can be the sample rate of biometric device (that is, biometric is set
The standby rate that data are provided).In other cases, the sample rate that platform uses can with receive the rate of data (for example, fixed
Rate, configurable rate etc.) it is unrelated.For example, if biometric device is configured as with 60 samples (60spm) per minute
Rate provides biometric data, then platform still can store data with the rate of 30spm.In other words, for 30spm's
Sample rate, platform will store five values after ten seconds, wherein and the first value is the second value sent by biometric device, second
Value is the 4th value sent by biometric device, and so on.Alternatively, if biometric device is configured as only working as biology
Continuous data provides biometric data when changing, then platform still can store data with the rate of 30spm.In such case
Under, the first value of platform storage can be the first value sent by biometric device；If the biometric device in storage
New value is not sent, then the second value of platform storage can be the first value sent by biometric device；If in storage
Biometric device is just sending a new value, then the third value of platform storage can be second sent by biometric device
Value, and so on.
Once video and biometric data are stored and linked, algorithm can be used to show the data together.Example
It such as, can be with if biometric data is stored (sample rate can be fixed or can configure) with the sample rate of 30spm
Start to show the first biometer magnitude (for example, being below video data, folded in video clipping using algorithm (for example, 216)
It is first-class to be added in video data), (two seconds in video clipping) the second biometer magnitudes of display, (video after two seconds after two seconds
Four seconds in editing) display third biometer magnitude, etc..In alternative embodiments of the present invention, non-video data (for example,
Biometric data, self-report data etc.) it can together be deposited with multiple timestamps (for example, each stamp of each storage value or offset)
Storage, timestamp can be together with the time starteds for synchronizing (for example, linking in time) different non-video data and/or regarding
Frequency evidence.
It should be appreciated that although client platform can be configured as automatically operation (that is, independently of master network device
240), but in one embodiment of the invention, certain functions of client platform are executed by master network device 240, and only
It can be executed when computing device 200 is communicated with host computer device 240.The advantages of such embodiment, is that it not only will be certain
Function is unloaded to host computer device 240, and ensure these functions can only be executed by host computer device 240 (such as, it is desirable that user
Cloud service is subscribed to execute certain functions).The function of being unloaded to cloud may include by non-video data together with video data
Function necessary to display is (for example, by Info Link to video data, by Info Link to non-video data, by non-video number
According to being synchronized to video data etc.), or may include more advanced function, for example, generating and/or shared " the excellent collection of choice specimens ".
In alternate embodiment, as long as computing device 200 is configured as meeting certain standards and is carried out aforementioned function.The standard may include
Computing device 200 is being communicated with host computer device 240 or computing device 200 is previously being communicated with host computer device 240
And the period since last time communicates is equal to or less than scheduled time quantum.Technology (example well known by persons skilled in the art
Such as, (allow the meter using the time of the authentication code (HMAC) based on hash method of encrypting key, the last time communication of storage
Calculate equipment determine the increment (delta) whether be less than predetermined time amount) etc.) and can be used to ensure that allowing execute specific function
The standard is satisfied before.
The block diagram of exemplary computer device and example biometric equipment is shown in FIG. 5.Specifically, exemplary
Biometric device 500 includes for sensing the sensor of biometric data, for interacting and showing various letters with user
The display of breath (for example, biometric data, setting data, operation data (for example, beginning, stopping and pause etc.)) is used for
It stores the memory of sensed biometric data, the transceiver for being communicated with exemplary computer device 600 and is used for
The processor of operation and/or driving transceiver, memory, sensor and display.Exemplary computer device 600 include for from
Example biometric equipment 500 receives biometric data (for example, using telemetering, any WiFi standards, DNLA, Apple
It is AirPlay, bluetooth, near-field communication (NFC), RFID, ZigBee, Z-Wave, thread, honeycomb, wired connection, infrared or other
Any one of the method for data transmission, data broadcasting, steaming transfer etc.) transceiver (1), for storing biometric data
Memory, for interact and shows with user various information (for example, biometric data, be arranged data, such as start,
Stop and the operation data of pause, the annotation in input session or addition speech explanation etc.) display, defeated for receiving user
Enter data (for example, self-report data) keyboard, for via internet (for example, using telemetering, any WiFi standards, DNLA,
Apple AirPlay, bluetooth, near-field communication (NFC), RFID, ZigBee, Z-Wave, thread, honeycomb, wired connection, it is infrared,
Or any one of the method for other data transmissions, data broadcasting, steaming transfer etc.) to host computer device provide biometric number
According to transceiver (2) and for operating and/or driving transceiver (1), transceiver (2), keyboard, display and memory
Processor.
The keyboard in keyboard or biometric device 500 in computing device 600, can be used for key in self-report data,
Or about user specific time how are you feeling today data.For example, if user feels fatigue, user can tap on keyboard
" T " button.If user feels that their endorphin works, user can tap " E " button on keyboard.And
And if user cheers up again, user can tap " S " button on keyboard.Then, which is stored and is linked
To sample rate (e.g., biometric data) or time stamp data, time stamp data can be the time that each button is pressed or
With the offset of time started.This will allow self-report data synchronous with video and/or biometric data (or linking in time).
It will also allow self-report data searched or filtering in order to find (for example, correspond to particular event (for example, when user starts
Feel fatigue) video, biometric or environmental data).
It should be appreciated that the present invention is not limited to block diagrams shown in fig. 5, also, it include the biometric of less or more component
Equipment and/or computing device are within the spirit and scope of the present invention.For example, do not include display or including camera and/or
The biometric device of microphone within the spirit and scope of the present invention, as except keyboard other data input devices or side
Method is (for example, touch screen, digital pen, voice/Auditory identification equipment, gesture identification equipment, so-called " wearable " or this field skill
The commonly known any other identification equipment of art personnel) like that.Similarly, only include a transceiver, further include that camera (is used for
Capture video) and/or microphone (for capturing audio or for being held by recording or measuring sound and sound how to propagate
Row spatial analysis) or further include the computing device of sensor (referring to Fig. 4) within the spirit and scope of the present invention.Should also
Understand, self-report data is not limited to user's how are you feeling today, but can also include the event that memory it is expected in user or application.For example,
User may wish to record (or adding timestamp) user cycling and pass by wild animal or specific building structure, or application may
Want record (or adding timestamp) patient and presses " request nurse " button or the non-biometric of any other user sensed
Activity.
Referring back to Fig. 1, as discussed above in conjunction with Fig. 2 B, main application (or client platform) can be in computing device
It is run on 108.In this embodiment, computing device 108 (for example, smart phone) can be configured as from biometric device
110 reception biometric datas (in real time or in later phases, utilize the time of the appearance corresponding to biometric data
Stamp), and make biometric data with record by computing device 108 (or the camera and/or microphone operated on it) oneself
State data, video data and/or audio data synchronizes.It should be appreciated that in this embodiment of the invention, in addition in local (example
Such as, on computing device 108) operation main application except, main application (or client platform) runs as previously mentioned.
Referring again to FIGS. 1, in another embodiment of the invention, computing device 108 further includes for sensing biometric
The sensor of data.In this embodiment of the invention, main application (or client platform) as previously discussed (counting by operation
Calculate local runtime in equipment 108), and act on at least synchronization video, audio, readme, environment and/or biometric data,
And allow synchronous data be played or be presented to the user (for example, via display portion, via being directly connected to computing device
Display equipment, via the user calculating equipment, etc. for being connected to (for example, directly, via network etc.) computing device).
It should be appreciated that in any embodiment, the present invention is not limited to Fig. 1 and computing device shown in Fig. 2 (quantity or class
Type), and may include any those skilled in the art it is widely known there is calculating, sensing, digital record, GPS or in addition
Positioning function equipment (for example, using WiFi positioning systems " WPS ", or export geographical location other forms (for example, logical
Cross network triangulation (triangulation))), for example, personal computer, server, laptop computer, tablet computer,
Smart phone, cellular phone, smartwatch, active belt, heart rate band, mattress sensor, sole sensor, digital camera, near field
Sensor or sensor device etc..It is also understood that the present invention is not limited to any specific biometric devices, and include by with
Be set to be worn in wrist (for example, as wrist-watch) biometric device, wear on the skin biometric device (for example,
As dermal patch) or the biometric device that is worn on scalp or computing device is incorporated into (for example, smart phone
Deng) in biometric device, be integrated or be added to article (for example, bedding, wearable device are (for example, clothes, shoes
Class, the helmet or cap or earphone or sports equipment (for example, racket, golf club or bicycle))) on biometric set
It is standby, wherein other kinds of data can also synchronously be measured with biometric, and and audio video synchronization, other kinds of data
Including physical property measure, for example, racket or club head speed or pedal rotation revolutions per second or footwear record such as impact zone,
Gait or shear force.Other data can also with video data synchronization be measured (for example, by least one sensor) or key
Enter (for example, passing through user), other data include animal biometric data (for example, in bull riding activity bull plus
Speed or rotary motion or the suddenly back of a bow jump up, the acceleration etc. of the horse to match with heart rate in horse racing), environmental data (example
Such as, the temperature at the position, position of user, position height above sea level, time of day etc.) and lifeless object physical property degree
Amount, for example, rev/min (for example, in such as vehicle of automobile, motorcycle etc.), miles per hour (suchlike) (for example,
In the such as vehicle of automobile, motorcycle etc., bicycle etc.) or G power (for example, by warps such as user, animal and lifeless objects
The G power gone through).As set forth above, it is possible to using sample rate and/or at least one timestamp, all data (are referred to as " non-to regard
Frequency evidence " may include metadata or the data about non-video data) it is synchronized to video data.
It is also understood that the present invention need not be with network (for example, internet) binding operation.For example, as shown in Figure 2 A, it is raw
Object measuring equipment 110 (itself such as can be the wireless activity band for sensing heart rate) and computing device 108 (itself for example, can
To be digital video recorder) the main host computer device 106 using (not shown) of operation is can be directly connected to, wherein main application
It acts on as previously mentioned.In this embodiment, video data, audio data, self-report data, environmental data and/or biometric
Data can (i) in real time or (ii) is provided to main application in later time because data are and sample rate and/or time
Stamp synchronization.This will allow for example, at least sportsman or sportsman or sportswoman (for example, gridder, football
Sportsman, racing driver etc.) video and action in sportsman biometric data (for example, see Fig. 7) together in action
Display (for example, play rugby, play soccer, racing car etc.).Only as an example, this will allow user in football player's dribble, kick
Whens ball, heading etc., the heart rate 730 of football player is watched.This can be completed by following：Usage time stamp 720 (for example,
Time started etc.) or using other sort methods using metadata (for example, sample rate etc.), make video data 710 and biology
Continuous data 730 synchronize, to allow user viewing specific time 740 (for example, 76 seconds) football player and with
The associated biometric data of sportsman of the specific time 740 (for example, 76 seconds).Similar technology is displayed for closing
In the biometric data of other sportsmen, card player, performer, game on line player etc..
In the case where it is expected to monitor or watch more than one individual from camera view, for example, being seen from remote care site
Examine the TV of the patient in hospital ward or the competitive sports (there are several sportsmen on sports ground) in such as football game
During broadcast, system can be configured so that, (in some cases, by using bluetooth or other wearable or NFC sensors
Its sensing function is also to have positioning function, to identify which particular individual tracked) examinee, sensor can be
It is feasible apart from it is upper send they biometric (if necessary, then combine relaying or beacon) so that viewer can regarding
While frequency or broadcast, switch the selection to track which or which individual biometric data, also, if it is desired to
It is possible if wanting and within the limitation that the video of used video camera captures field, then also so that the view of video camera
It concentrates on the group reduced or specific individual.In alternative embodiments of the present invention, in the video frame based on such as individual
Position (for example, center of frame), rate travel (for example, than other individual quickly move) or with sensor (for example, by a
Body is worn, the embedded ball carried by individual is medium) degree of closeness, the selection of biometric data can be automatically completed,
Middle sensor can be previously active or by remote radio frequency signal activation.The activation of sensor can lead to the biometer of individual
Amount data are sent to receiver, or can allow receiver in other data just sent (for example, from other individuals
Biometric data) in identification individual biometric data.
In the background of body-building or motion tracking, it should be understood that capture the activity of individual on video independent of third
This point is accomplished in the presence of side, it is contemplated that the method for various self-timers, for example, mounted on the wrist or body of examinee
On safety belt (harness) or on self-timer attachment or gimbal or fixed to the video capture device (example on object
Such as, sports equipment (for example, bicycle handle bar), the object that is found in Sports Environment are (for example, basketball or tennis net, football gate
Column, ceiling etc.), the unmanned plane camera, the tripod etc. that follow individual).It should further be noted that such video capture device
May include more than one camera lens so that can not only record a video to the activity of individual, but also can be different simultaneously
View, for example, individual the front watching or (that is, ambient enviroment of user) for seeing.Video capture device
It may be fitted with convex mirror lens or using convex mirror lens as attachment addition before camera lens or be complete 360 degree of phases
Machine or the multiple 360 degree of cameras linked together so that use or do not use specific software as known in the art, all
360 degree of panoramas can be generated or surround view, or 360 degree of global views on all axis can be generated.
Under the background of augmented reality or virtual reality, individual wears the augmented reality (" AR ") properly equipped or virtual existing
Real (" VR ") glasses, goggles, earphone, or equipped with 3D rendering that can render AR, VR or other synthesis or true its
The viewing display of his type, individual can watch biometric data (for example, heart rate) from sensor, together with from
Other data for the sensor (for example, can measure the accelerator of movement and speed) properly equipped are (for example, exercise stroke or speed
Degree), wherein the data investigation is on their visual field.In addition, the incarnation of the individual in movement can be superimposed upon regarding for individual
Wild front so that they can monitor or improve their athletic performance, or by watching themselves or themselves
Incarnation, together with (for example, synchronous) their performance (for example, biometric data), carry out the experience of lifter motion.Optionally,
It can also show the biometric data of their incarnation simultaneously in the visual field or compete the biometric data of incarnation.In addition
(or alternatively), at least one additional training or competition incarnation can be superimposed upon on the view of individual, this can show with
Individual it is relevant competition incarnation (for example, show they be superimposed upon before individual, show they be added to user side, show
Go out them and is added to behind user (for example, in rearview mirror part of display etc.), and/or shows that they mutually see with individual
(for example, luminous point (blips) etc. in the radar screen portions of display) etc.).Can use any such as their friend or
Person trains the competition incarnation of the true people of acquaintance etc, to encourage user to improve or correct their performance and/or make it
Exercise habits it is more interesting (for example, by allow individual take exercise or training when in AR, VR or mixed reality (" MR ")
Environment in " competition ", alternatively, either (destination or position can be the imaginations for position by visualizing virtual destination
Either true can scan or synthesized by computer modeling, for example, historical place) virtually " gameization "
Their activity).
In addition, any multimedia sources that user is exposed in the activity for participating in just being tracked and recording, it should be similar
Ground can utilize timestamp to record, the analysis for the biometric response of individual and/or correlation.The example of this application can
To be for the selection when someone is being trained activity to specific music track, wherein individual is past to be based on example
As heart rate (and how they reach specific performance level or target) response with music type (for example, specific music is bent
Mesh, similar to specific music song music track, by listened or liked specific music song other people recommend or selection
Music track etc.) correlation, be used to develop personalized algorithm, to optimize automation the selection of music, with enhancing muscle power,
Or maximumlly restore during exercise and after taking exercise.Based on personalized selection algorithm, individual can also specify them
Specific music song or music type are wished based on their geographical location to play；Such example will be frequently or periodically
Ground using particular electrical circuit for train or entertain purpose someone.Alternatively, can be rung by recording or being associated with past biometric
It answers, in conjunction with the readme input when listening to particular track, to select the song or type of music.
It should be appreciated that biometric data need not be associated with body kinematics or sports, but can be with fixation
The video combination of individual at position (for example, for subsequent review, the individual position for being just remotely monitored or recording), example
Such as, as shown in figure 3, for healthy reason or medical conditions (for example, in their family or hospital) or assisted living ring
The elderly in border or the baby to sleep monitored in another room or another location by parent.
Alternatively, individual may be driving through park or shopping center, or near park or shopping center,
In, their position usually is recorded by adding geography to stab (geo-stamping), or by adding geographical stamp (for example, in certain bits
The height above sea level or weather set) addition additional information and the information and content what is (its by individual internet or intelligence/
Watch or interact on the TV of enabling or on any other networked devices comprising screen) (for example, specific wide
Announcement, movie trailer, appointment profile etc.) and can watch or record by video they with this information and content
It interacts, in conjunction with their biometric data, all these data sources can be by synchronous for looking back, and it is by each to synchronize
These individual sources are added timestamp or suchlike (for example, sampling etc.).This will allow third party (for example, service carries
For quotient, advertiser, advertising provider, studio/promoter, date profile poster, meet-a-mate site etc.) obtain and see
Crowd's viewing associated biometric datas of specific data, to analyze their response, wherein spectators or their profile can be with
Selectively by third-party system identification, or it is known or can wherein there was only the identity of the interactive device of spectators
To be obtained from the GPS of biometric sender (or other with positioning function) equipment.
For example, advertiser or advertising provider can see how people respond advertisement or studio/push away
Wide person can assess how people respond movie trailer, or the poster of appointment profile or meet-a-mate site itself can see
People introduce appointment and how to respond.Alternatively, game on line or eSports broadcast services (for example, twitch.tv) or by electricity
Depending on the viewer of the online player of online canaster broadcasting or that stream is broadcast, the biometric for enlivening participant can be watched
Data, while main view frequency source and the visual response or performance of participant can be watched.As video/audio, this can be with
It synchronizes using (for example, sample rate etc.) real-time synchronization such as embedded timestamp or later.In addition, generating face from source video
In the case of Expression analysis (for example, measuring individual under the background of the response of advertisement information), since video is added
Timestamp (for example, utilizing the time started), so facial expression data can be synchronized to the physics biometric data of individual,
And it is associated with the physics biometric data of individual, individual physics biometric data has similarly been added timestamp
And/or sampling.
As discussed previously, main application, which can be configured as, executes multiple functions.For example, main application can be configured as
Keep video and/or audio data synchronous with biometric data.This will be so that such as (such as on TV, computer screen etc.)
How the biometric data that the individual of viewing competitive sports can watch each player changes during the broadcasting of competitive sports,
Or the change of these biometric datas can also be mapped to other players or other comparison models.Similarly, doctor, shield
Scholar or Med Tech can while the biometric data of guarder recorder sleep habit, and watch, search for or
The record (for example, on TV, computer screen etc.) is looked back later.The system can also be using machine learning come for each trouble
Person builds profile, identifies the special characteristic (for example, their heart rate rhythm, their breathing pattern etc.) of patient, and work as and surveyed
The feature of amount notifies doctor, nurse or Med Tech or triggering alarm when occurring abnormal or irregular.
Main application can be additionally configured to provide biometric data to remote user via the network of such as internet.Example
Such as, the biometric device smart phone of blood alcohol sensor (for example, with) can be used for measuring the blood alcohol of people and contain
(for example, when people is talked by smart phone and remote user) is measured, and the blood alcohol content of the people is provided to remote user.
By placing a sensor near microphone or sensor being incorporated in microphone, such system will be so that parent can be with
By participating in determining whether their child is drinking always with the phone of their child or video call.It is known in the art
Different sensors can be used for sense people breathing in different chemical substances, or by analyze sound and speed variation
To detect the breathing pattern of people so that monitoring side can determine whether examinee uses always alcohol or other controlled substances, or
Person is in order to which other Diagnosis of Primary are because carrying out breast rail.
The system can also adapt to integrated so-called " chip lab " (LOC) in equipment itself, or adaptation
In the suitable attachment for being added to the remote testing for being used for such as blood sample thereon, wherein smart phone is used for the acquisition of sample
With the transmission to test laboratory for analysis, or for carrying out sample collection and analysis in equipment itself.It is any wherein
In the case of, system is adapted such that by acquiring and/or submitting the point of sample while the video that adds timestamp of record and right
Between the sample added-time and/or position (or GPS) is stabbed, for the purpose of sample and analysis integrality and patient identity certainty, to quilt
The identity of inspection person and their blood sample carry out cross-certification.This confirmation to identity is for the prison under tele-medicine background
Pipe, record preserve and health insurance reason is especially important, because individual will fulfil function as follows, the duty more and more
It can be up to now usually at related facility scene by thering is qualification and defined medical treatment or lab assistant to execute, rather than by quilt
Inspection person is executed using networked devices, either for uploading to central analysis facility, or for long-range point to equipment itself
Analysis.
The set of this or other biological continuous data (such as heart rate or blood pressure) can also be applied to following situation：For
Security reason and it is essential that, by it is real-time periodically remote video monitoring come verify whether for example the pilot of aircraft,
Truck or engine driver be in the situation of health and sound (sound) with their vehicle of control or ship or for example whether
They are undergoing unexpected incapacitation or heart attack etc..Because monitored people is on added timestamp with offer plus ground
Reason stamp and/or sampling the biometric data identical time is videoeding, so monitored people or third party " takes advantage of
Deceive ", " personation " or the possibility smaller around system.In patient/doctor's telereference setting, which can be used for safety
Video consultations also verify the consulting wherein from the perspective of regulation or health insurance by time and/or geographical stamp verification
And its occur.In addition, in the case where needing the certification of higher level, system can also be adapted to use face recognition or life
Object measures algorithm, to ensure that correct people is monitored or facial expression analysis can be used for behavior pattern evaluation.
Monitored side can be mitigated by being incorporated to various supplementary features to be not intended to by persistently monitoring (for example, the elderly is not
Want to allow they each action and action continuously videoed) worry.In one embodiment, video will be enduringly recorded in
In the circulatory system using the memory space of reservation, in predetermined time segment record, then the auto-erasing video, wherein n are indicated
Minute selected in cycle, E are events as follows, prevent the cycle of the n minutes of record to be wiped free of, and trigger quilt
The real-time ability for sending and refunding (rewind) of the visible state of guarder or action extremely monitoring side so that monitoring can
It is enough to look back the physical manifestations for leading to E.The trigger mechanism of E can be the hair of the biometric data for example except predefined scope
It gives birth to or such as by movement or position sensor (for example, in the healthy carrying device that the elderly is worn or in their shifting
On mobile phone or the motion sensing device of other networkings near it, such as gyroscope, accelerometer or magnetometer) activation
Such as decline other abnormal notices of (fall) alarm.Monitoring side is monitored the body of side by that can not only be checked after E
State, while obtaining reading while its associated biomolecule continuous data, but also can look back and directly result in event trigger notice
Event and biometric data.Alternatively, it can further calibrate so that regarded although being recorded in n cycles as previously described
Frequently, but until the generation of E, the video recycled from n is actually just sent to monitoring side.The advantages of system includes
In terms of body privacy, wherein third party may only obtain the time before critical event and event, to also lead to necessary transmission
The expectation optimization of both bandwidth and call data storage.It should be appreciated that above system can also be configured such that for long-range old
The E notices of year people, baby or patient monitoring are further adapted to include feature tracking and/or Expression Recognition feature.
If the video data and biometric data of user be stored in themselves equipment by themselves or
Outside themselves or on the safe third party of oneself " cloud " storage device, but (it makes the index metadata of source material
Source data can be ranked up, extrapolated, searched for and generally handled) it is retained in central server (for example, in medical records
In the case of, it is retained in office or other medical facilities of doctor), then privacy can be further increased for user.It is such
System will so that monitoring side can when seeking advice from Internet access video and other data, but video etc. is still gathered around by examinee
Have.Except the privacy and their Information Security of enhancing user, by the trustship of the storage of video and biometric source data
(host) being separated with the processing of data further advantages in that, due to its storage for examinee local, it is not necessary to upload it and arrive
Calculation server not only to reduce cost, but also improves storage efficiency and data bandwidth.This will be had in the case where there
Benefit：For the long-distance Log server for this test that the qualified medical worker at the position different from examinee looks back, hair
Life is in the region that low-bandwidth network covers.It can also be selected to reduce the frame rate of audio-visual-materials, as long as making it and adopting
Sample rate one, which is shown, confirms correct timestamp (as previously described).
It should be appreciated that by storing information at central server (or main equipment), it is possible to implement as is generally known in the art
Various technologies carry out protection information, and prevent it is unwarranted individual or entity access information.Thus, for example, can be to user
(or user is allowed to create) user name, password and/or any other identification (or certification) information are provided (for example, user biological meter
Amount, key card (fob) etc.), and main equipment can be configured as and permit (grant) to letter using identification (or certification) information
Cease the access of (or part of it).It can be to safe mistake as third party's implementation of class of such as medical supplier, insurance company etc.
Journey, to ensure that information can only be accessed by authorized individual or entity.In a particular embodiment, certification can allow to access all
The only a part of the data of storage or the data of storage is (for example, user authentication can allow to access personal information and storage
Video and/or biometric data, and Third Party Authentication can be only to allow the video and/or biometric data of access storage).
In other embodiments, certification is for determining that the individual for logging on to main equipment or website or entity can be serviced with.For example, net
Video/audio data can only may be synchronized to biometric data and/or executed just by the visitor (or non-subscribed person) to stand
Grade (rudimentary) search or other processing, and video/audio data can be synchronized to biometric number by subscriber
According to and/or execute more detailed search or other processing (for example, creating excellent collection of choice specimens etc.).
It is also understood that although index metadata is only maintained at central service with data upload efficiency in order to store
Device thus provide for different types of data interoperability common platform and the equipment of user oneself (for example,
iCloudTM、DropBoxTM、OneDriveTMDeng) on storage video and/or audio data be advantageous, but the present invention it is unlimited
In this.Therefore, in a particular embodiment, in the case that feasible, it may be advantageous below：(1) it stores on a user device
Data (for example, video, audio, readme, biometric data and metadata) (such as so that user equipment can be independently of master
Equipment operation)；(2) data are stored on central server (for example, main equipment) (for example, video, audio, readme, biometric
Data and metadata) (for example, user is allowed to access data from any equipment with network function)；Or (3) are set in user
Standby upper storage first part (for example, video and audio data), and store second on central server (for example, main equipment)
Partly (for example, biometric data, self-report data and metadata) (such as so that user can be in user equipment and main equipment
Synchronous video/audio/biometric data is only watched when communication so that user can be when user equipment be communicated with main equipment
It searches only for biometric data (for example, to create " the excellent collection of choice specimens ") or biometric data is ranked up (so as on time
Between sequence, size (from up to minimum), size (from most as low as highest), best comment, worst comment, at most viewing, minimum
Viewing etc. identifies and/or lists data), etc.).
In another embodiment of the invention, the function of system further (or alternatively) by user equipment and/or
The software limitation operated on main equipment.For example, the software operated on a user device can allow user to play video and/or sound
Frequency evidence, but do not allow user that video and/or audio data are synchronized to biometric data.This may be because of center clothes
Business device be used to store to the data (timestamp index, metadata, biometric data, sample rate etc.) of sync key and/or
The software operated on host equipment is necessary for synchronizing.As another example, the software operated on a user device can be with
User is allow individually or with biometric data synchronously to play video and/or audio data, but perhaps so that using
Family equipment cannot (or can be with ability of restricting user equipment) search biometric data or otherwise from biometer
It measures and biometric data is extrapolated or handled in data, to identify relevant portion (for example, it can be used for creating synchronous video/sound
Frequently " the excellent collection of choice specimens " of/biometric data) or sort to biometric and/video data.This may be because of central service
Device is used to store data (biometric data, the biometric metadata for biometric data key of searching for and/or sort
Deng) and/or software be for search (or execute Advanced Search) and/or sequence biometric data (or executing advanced sequence) must
It needs.
In any or all above-described embodiment, system can also be adapted for include password or other forms certification, with
Make it possible to be had secure access to (or refusing the unwarranted access of absolute data) to data in one or both directions, make
The data for obtaining user's access host or host access user need to permit (permission).User and monitoring side or host it
Between interaction occur in real time (for example, between patient and its healthcare practitioners or other medical workers security video consulting
In) when, it can exchange and watch data by establishing Virtual Private Network (VPN).Alternatively or it is possible to further
At data source (such as at storage device of individual, either local storage or storage device based on cloud) and/
Or at monitoring review side (for example, during the patient at medical facilities records, or in main management rank) (raw for real data
Object metering, video, index of metadata etc.) encryption.
Under the background of the monitoring of very immature baby, a crucial and general unaccountable problem is sudden
Infant death syndrome (SIDS).Although the incidence of SIDS is generally without method interpretation, various equipment attempt to prevent its generation.
However, by will in infanette or nearby include measure relevant biometric data (including heart rate, sleep pattern,
Breathalyser and such as environment temperature other measurement) sensor device current system element, with when predefined
Between capture movement, audible breathing or lack exercise or the recording equipment of audible breathing (that is, silent) is combined in section, can
Various parameters are arranged in conjunction with the videograph added timestamp by parent or other monitoring sides, to provide more fully alarm,
It more timely acts or intervenes to initiate user, or judgement does not need action response actually really.In addition, in such as baby
In the case of bed monitoring situation, system can be configured as to be opened from previous observation (with or without the input from monitoring side)
Learning algorithm is sent out to help to distinguish that is " normal ", what is false positive or what may be constituted exception, and is therefore exhaled
It sighs taking action.
Main application can be additionally configured to play video data synchronous with biometric data, or search for specific
Biometric data presence.For example, it is as discussed previously, the videograph with sound is carried out by sleeping to people, and
And keep record synchronous with biometric data (for example, sleep pattern, brain activity, snoring, breathing pattern etc.), it may search for giving birth to
Object continuous data is to identify that it is more than threshold where specifically to measure (sound level or silence period that such as, such as with decibel measure)
It is worth or drops to threshold value or less so that doctor, nurse or Med Tech can watch corresponding video section, should without watching
The entire video of people's sleep.
This method is shown in FIG. 6, since step 700, wherein receiving (or link) biometric number in step 702
According to time stamp data (for example, time started, sample rate).Audio/video data and time stamp data (such as the time started
Deng) and then be received (or link) in step 704.Then usage time stamp data (coming from step 702 and 704) make biometric
Data are synchronous with audio/video data.Then allow user's operation audio/video in step 708.If user selects to play,
Then audio/video is played in step 710.If user selects search, so that user may search for biometric in step 712
Data.Finally, if user selects to stop, stopping video in step 714.
It should be appreciated that the present invention is not limited to steps shown in Fig. 6.For example, allowing user to search for meets at least one
The biometric data of part, the corresponding part (or just part before the condition) for playing video and in biometric
(or just after biometric data no longer meets the condition) stops broadcasting after data no longer meet at least one condition
The method for putting video is within the spirit and scope of the present invention.As another example, if this method is related in user equipment
Interacted between main equipment so that video/audio data it is synchronous with biometric data and/or search biometric data,
Then this method may also comprise the following steps:：Biometric data and/or metadata are uploaded to main equipment (for example, in the implementation
In example, video/audio data can store on a user device)；And it is created using biometric data and/or metadata
Related or significant data are searched for for synchronous timestamp index, and/or in biometric data (for example, being more than threshold
The data etc. of value).As another example, if audio/video data and biometric data play together in real time (synchronous),
Or (for example, in step 710), then this method can not need step 706 for broadcasting in data playback.
In one embodiment of the invention, as shown in figure 8, may also comprise the video data 800 of audio data in the time
" T " starts and continues the duration " n ".Video data is stored preferably in memory (locally and/or remotely) and chain
Other data are connected to, for example, identifier 802, time started 804 and duration 806.Such data bind video data
To at least special session, specific time started, and identify including video duration.In the implementation of the present invention
In example, each session may include different activity.For example, going to the destination positioned at Berlin or following the spy in particular day
The travelling (session) for determining stroke may include cycling across city (the first activity) and walk through park (the second activity).
Therefore, as shown in figure 9, identifier 802 may include the session for uniquely identifying session by globally unique identifier (GUID)
Identifier 902 and movable activity identifier 904 is uniquely identified by globally unique identifier (GUID), wherein session/work
Dynamic relationship is the relationship of father/son.
In one embodiment of the invention, as shown in Figure 10, biometric data 1000 stores in memory simultaneously chain
It is connected to identifier 802 and sample rate " m " 1104.This allows biometric data to be linked to video data in playback.Example
Such as, if identifier 802 is 1, the time started 804 is 1:00PM, video duration is 1 minute, and sample rate 1104 is
30spm, then 2:00PM, which will play video, to be caused：First biometer magnitude (biometric (1)) will be 2:00PM shows (example
Such as, below video, it is first-class in video), to show (for example, below video, first-class in video) second biometer after 2 seconds
Magnitude (biological characteristic (2)), and so on, until video is 2:01PM terminates.Although self-report data can be as biometric number
It is still, more advantageous if such data are only periodically received according to like that by storage (for example, being linked to sample rate)
Be to store the data 110 as shown in figure 11, that is, be linked to identifier 802 and timestamp 1104, wherein " m " be receive from
It states the time of (or from realize) data 1100 or is offset between the time and time started 804 (for example, the time started
10 points 4 seconds etc. afterwards).
This can see that wherein video data starts in time T in such as Figure 14, and biometric data is adopted for every two seconds
Sample (30spm), and self-report data is received in time T+3 (or by 3 units of time started).When video 1402 is being broadcast
When putting, the first biometer magnitude 1404 is shown in time T+1, shows the first self-report data 1406 in time T+2, and in the time
T+4 shows the second biometer magnitude 1406.By storing data in this way, video and non-video data can be separated from each other
Storage, and synchronize in real time or when playing video.It should be appreciated that although data are stored separately for minimum memory
It may be advantageous for the equipment of device and/or processing capacity, but client platform can be configured as create new video data,
Or the data of the video and non-video data including simultaneous display.Not only it can use single " the excellent collection of choice specimens " (single file) will
Different non-videos and video data link together (in relationship), so that social media website can be used (all
Such as FacebookTMOr YoutubeTM) shared file, and use standard playback software (such as QuicktimeTM) played file.Such as
It is discussed more fully below, the excellent collection of choice specimens may include the various parts (or editing) of video data (for example, when specific work occurs
When dynamic etc.) and corresponding biometric data.
When then display sampled data, client platform can be configured as shows the number using specific extrapolation technique
According to.For example, in one embodiment of the invention, as shown in figure 12, the first biometer magnitude 1202 is shown in T+1, it is aobvious in T+2
Show the second biometer magnitude 1204, third biometer magnitude 1206 shown in T+3, can use known extrapolation technique (including
Linear and nonlinear interpolation (interpolation) and all other extrapolation generally known to those skilled in the art and/or
Interpositioning) in non-sampled time showing biometric data.In another embodiment of the present invention, as shown in figure 13,
One biological variable 1202 retains over the display, until showing the second biometer magnitude 1204, the second biometer magnitude
1204 retain over the display, until showing third biometer magnitude 1206, etc..
About by data link to the identifier that can be linked to other data (for example, time started, sample rate etc.), such as
Fruit data can be linked to the identifier of current sessions (and/or activity) by real-time reception, then data.However, when in the fact
When (for example, after conversation end) receives data later, there are data can be linked to special session and/or activity (or with
Its associated identifier) several ways.Data can link (for example, by user) or manually by applying AutoLink.
For the latter, this can for example be completed by following：By duration (for example, video length) participant of received data
Words and/or movable duration are compared；Assuming that received data is related to nearest session and/or activity；Or
Analysis includes the data in received data.For example, including the data in the data received in one embodiment
(for example, metadata) can identify time associated with data and/or position, the data-link that then can be used for receive
It is connected to session and/or activity.In another embodiment, computing device can show or play identification session and/or movable
Data (for example, bar code (such as QR codes), sound (such as repetitive sequence of note) etc.).External video/voice-frequency sender can
With together with user and/or his/her environment (for example, before it, later or period) record identification data (such as by calculating
What equipment showed or played).Using then may search for video/audio data to identify data, and will be regarded using the data
Frequently/audio data is linked to session and/or activity.If desired, the identification division of video/audio data then can be by
Using deletion.It in alternative embodiments, can be in physical equipment (for example, can allow to through network (for example, via intelligence
Can phone) carry out medical data communication medical test module) on printing bar code (for example, QR codes), and bar code can
The data provided with equipment for the video of the synchronous user using equipment (as previously described).The medical test module the case where
Under, the video that the printing bar code in module may be used to test is synchronous with the test result that module provides.At another
In embodiment, computing device and external video/voice-frequency sender all are used to record the video and/or audio of user (for example, user
Statement " start Berlin and ride a bicycle session " etc.), and the data provided using user by video/audio data to be linked to session
And/or activity.For example, computing device can be configured as the data link that provides user to special session and/or activity (example
Such as, having begun, will the starting, session that just terminates and/or activity, etc.), and be configured with video/
The data that user in audio data provides, special session and/or activity are linked to by video/audio data.
In one embodiment of the invention, client platform (or application) is configured as calculating in smart phone or tablet
Hands- operation.Platform can be configured as establishment session (individually or together with the software operated on the host device), in meeting
Video and non-video data are received during words, and video data (synchronously) is played back together with non-video data.Platform may be used also
So that user may search for session, certain videos and/or non-video event are searched for, and/or create the excellent collection of choice specimens.Figure 15-figure
29 show the exemplary screen shots of this platform.
For example, Figure 15 shows exemplary " logining " screen 1500, permission user logins application and Internet access is deposited
Store up that application on computing device and/or host computer device is relevant, the specific data of user.Login may include for answering
With, company's cloud or social interaction server website (such as FacebookTM) unique User ID and password.
Once user logins, user can be allow to be created as shown in figure 16 by exemplary " creating session " screen 1600
Build session.When creating session, user can be allow to select camera by video data is provided (for example, in computing device
Internal, outside computing device (for example, computing device can be connected to by wired or wireless by access to the Internet),
Etc.).Once having selected camera, the video data 1602 from camera can be displayed on the screen.It is also possible that user
Can select by provide biometric data biometric device (for example, inside computing device, outside computing device
(for example, computing device can be connected to by wired or wireless by access to the Internet), etc.).Once having selected biology
Measuring equipment, the biometric data 1604 from biometric device can be displayed on the screen.Then, user can lead to
Click " starting session " button 1608 is crossed to start session.Although selection processing preferably executes before the session start,
User can postpone the selection of camera and/or biometric device, until conversation end.This makes application that can receive not
It is real-time available data or by being not already connected to the equipment of computing device (for example, will be inserted into once conversation end
To the external camera in computing device) provide data.
It should be appreciated that in a preferred embodiment of the invention, clicking " starting session " button 1608 and not only starting instruction meeting
The timer 1606 of the current length of words, but also trigger storage in memory and be linked to the globally unique identifier of session
At the beginning of according with (GUID).By the way that video and biometric data are linked to GUID and GUID is linked to the time started,
Video and biometric data are also linked to the time started (according to definition).Other data of such as sample rate can also pass through
The data link is linked to biometric data to biometric data or by the data link to GUID, the wherein GUID
Then it is linked to biometric data.
Before the session start or after conversation end, as shown in figure 17, user can be allow to pass through example
Property " session name " screen 1700 key in session name.Similarly, as shown in figure 18, it is also possible that user can pass through example
Property " conversation description " screen 1800 key in conversation description.
Figure 19 shows exemplary " session start " screen 1900, is the screen that user may see when session is run
Curtain.On the screen, user can see video data 1902 (if providing in real time), 1904 (such as fruit of biometric data
When provide) and session current run time 1906.If the user desired that suspending session, then user can press " pause session "
Button 1908, alternatively, if the user desired that stopping session, then user can press " stopping session " button (not shown).Pass through
" stopping session " button (not shown), conversation end are pressed, and dwell time is stored in memory and is linked to session
GUID.Alternatively, by pressing " pause session " button 1908, time out (the first time out) is stored in memory
In and be linked to session GUID.Once pause, session can then restore (for example, by pressing " restore session " button,
It is not shown), this will cause recovery time (the first recovery time) to be stored in memory and be linked to session GUID.No matter can
Words be start and stop (that is, leading to single continuous videos), or start, pause (any number), restore (any number) and
Each beginning/time out stored in memory should all stored by stopping (that is, leading to multiple video clippings)
Corresponding stopping/recovery time is stored in device.
Once session has stopped, then session, such as Figure 20 can be looked back by exemplary " looking back session " screen 2000
It is shown.In its simplest form, review screen can play back the video data for being linked to session (for example, single continuous videos
If (session does not include at least one suspending/resuming), the multiple video clippings played one by one are (if session includes extremely
A few suspending/resuming) or multiple video clippings (if multiple video clippings are relative to each other) for playing together (for example, holding
Two videos (for example, from different visual angles) of the user of row specific activities, user execute spy while watching the second video
Fixed movable first video (such as training video)).If user wants to see the non-video number shown together with video data
According to then user can press " display graphical options " button 2022.By pressing the button, presented as shown in figure 21 to user
Illustratively " graphical display option " screen 2100.Here, user can select him/her to want to see together with video data
Data, such as biometric data (for example, heart rate, heart rate variability, user velocity, etc.), environmental data are (for example, temperature, sea
Pull out, GPS, etc.) or self-report data (for example, user is in ession for telecommunication how are you feeling today).Figure 22 is shown including video data
2202 and biometric data exemplary " look back session " screen 2000, biometric data can with graphically 2204 or
Written form 2206 is shown.If in video it can be seen that more than one individual, application can be configured as while
By user selectively (such as so that user can by select the first individual come look back about first individual biometric number
According to so that user can look back the biometric data individual about second by selecting the second individual) it shows about each
The biometric data of individual.
Figure 23 shows exemplary " map " screen 2300, can be used for showing GPS data to user.Alternatively,
(for example, below video data, first-class in video data) can be presented in GPS data together with video data.It can also be to user
Exemplary " summary " screen of the display session information (such as session name, conversation description, various measurements etc.) of session is presented
2400 (referring to Figure 24).
By being stored separately video and non-video data, data can be easy searched for.For example, Figure 25 show it is exemplary
" biometric search " screen 2500, wherein user may search for particular organisms metering value or range (that is, biometric event).
As an example, user may wish to jump to point of its heart rate between 95 to 105 times per minute (bpm) in session.Figure 26 is shown
Exemplary " the first result " screen 2600, the wherein heart rate of user enter session 42 seconds 20 minutes for 100.46bmp (such as
Referring to 2608).Figure 27 shows that exemplary " the second result " screen 2700, the wherein heart rate of user divide 57 seconds into session 23
For 100.48bmp (for example, see 2708).It should be appreciated that other events can be searched in a session, including Video Events and from
State event.
The data in session are not only may search for, but also may search for the data from multiple sessions.For example, Figure 28 is shown
Exemplary " session search " screen 2800, wherein user can key in specific search criteria, including session date, session are long
Degree, biometric event, Video Events, readme event etc..Figure 29 shows that exemplary " list " screen 2900, display meet
The session of the standard of key entry.
The present invention is described as personalization preferences optimization, or at least one emotion using individual (in the second portion)
State, mood, physical condition or the state of mind (" state ") (for example, the determination using the biometric data from individual,
Etc.) determine response, the response may include as at least one state result individually or with other data (for example,
It is undergoing at least one things (or relative data) of the time of at least one emotion near individual in individual
Deng) it is supplied to the data based on web of individual together.
As shown in figure 30, the preferred embodiment of the present invention is according at least with content supplier (for example, the data based on web
Provider) 3104 and the web host 3102 that is communicated by wide area network (WAN) 3100 of at least one network equipment 3106 operate,
Wherein each network equipment 3106 is by individual operations and is configured as the biometric data of individual being transmitted to web host
3102, wherein obtaining biometric data using at least one biometric sensors 3108.
Although Figure 30 describes preferred embodiment, it is to be understood that, other embodiments are also in the spirit and scope of the present invention
It is interior.For example, the network equipment 3106 can be configured as the biometric data of acquisition (for example, sensing etc.) individual in itself.This can
With for example by using at least one microphone (for example, with from individual place acquisition voice data), at least one camera (for example,
With obtain individual video data), at least one heart rate sensor (for example, with measure individual heart rate data), it is at least one
The completions such as respiration transducer (for example, to measure the breathing chemical composition of individual).As another example, host can be configured
For for example using the wireless protocols and network equipment direct communication of bluetooth, Wi-Fi etc..As another example, host can be with
Be configured with a for example, at least microphone, at least one camera or at least one sensor (for example, heart rate sensor,
Respiration transducer etc.) directly biometric data is obtained from individual.In this example, host can be configured as provides to individual
Data (for example, the display data on host display) or at least one action of execution (are driven automatically for example, automobile is switched to
Sail, limit speed etc.).
With reference to figure 30 and Figure 31, content supplier 3104 provides the data based on web to web host 3102, such as website,
Webpage, image data, video data, audio data, advertisement etc..It is other to be carried by least one other content based on the data of web
It is further provided to web host 3102 for quotient's (not shown).The multiple data based on web are (for example, multiple websites, multiple
Webpage, multiple images data, multiple video datas, multiple audio datas, multiple advertisements etc.) from such as by different biometrics
Other data (as described below) of the data link to the information of different conditions (referring to Figure 32) and interesting data (referring to Figure 33)
Together, it is stored in memory devices 3204.It should be appreciated that the present invention is not limited to memory devices shown in Figure 31
3204, and may include being located at the additional memory storage devices internally and/or externally of web host 3102 (for example, database
Deng).
Then, web host 3102 is configured as receiving biometric data from the network equipment 3106.As described above, biological
Continuous data is preferably related to the individual of the network equipment 3106 is being operated (that is, from its acquisition), and it is possible to using at least
One biometric sensors 3108, external heart rate sensor etc., connect the receipts biometric data.As described above, this
Invention is not limited to biometric sensors 3108 shown in Figure 30, and may include being located at outside the network equipment 3106
Additional (or different) biometric sensors (such, such as microphone, camera etc.), and/or set positioned at network
Standby internal at least one biometric sensors (such, microphone, camera etc.).If biometric sensors
Positioned at the outside of the network equipment, then it can be via at least one wiredly and/or wirelessly (for example, bluetooth, Wi-Fi etc.) and network
Equipment communicates.
It should be appreciated that the present invention is not limited to any certain types of biometric data, and may include for example a
Heart rate, blood pressure, respiratory rate, temperature, eyeball expansion (eye dilation), eye movement, facial expression, the voice fundamental of body
(speech pitch), the sense of hearing change (auditory changes), body kinematics, posture, Blood Hormones, urine chemistries
The measurable body or biological characteristic of concentration, breathing chemical composition, sialochemistry ingredient and/or any other type.Biology
Continuous data can be the variation (for example, variation of heart rate) of particular value (for example, specific heart rate etc.) or value, and can with it is more
It is related in one feature (for example, heart rate and respiratory rate).
Although being also understood that optimum from the direct measurement to known individual, identical correlation
(correlation) method can be applied to the people of general category.Another example is facial-recognition security systems it is known that：In spy
90% people that (for example, hospital) is set in positioning fears, and known individual is in the position.Even if the biometric of the individual
Data are not shared with system, correlation can also be applied, to protect privacy and still allow for statistically significant target alignment.
Another example is a bar, is integrated with urine chemistry system in the bathroom, to provide about the people's in bar
General information.Coordinated (coordinate) with time and position it is then possible to which the data are returned in lineup, and
Significant correlation for message to be aligned to individual (for example, in the individual in bar within this period) is provided.
As shown in figure 31, web host 3102 is at least one including being configured as determining from the biometric data received
The application 3208 of state.This using algorithm known and/or the biometric data being stored in memory devices 3204 from it is different
Correlation between state (such as, affective state) is completed.For example, as shown in figure 32, if biometric data 3302 refers to
Show that individual is being smiled (for example, by using at least one camera), then can determine that the individual is undergoing happy emotion
3304.As other examples, if biometric data 3302 indicates that the heart rate of individual is steadily increasing (for example, passing through
Use heart rate sensor), then it can determine that the individual is undergoing angry emotion 3304.If biometric data 3302 refers to
Show that the heart rate of individual is temporarily increased (for example, by using heart rate sensor), then it is surprised can to determine that the individual is undergoing
Emotion 3304.If biometric data 3302 indicates that individual is being frowned (for example, by using at least one camera), can
To determine that the individual is undergoing sad emotion 3304.If biometric data 3302 indicates that the nostril of individual is being opened
(for example, by using at least one camera) can then determine that the individual is undergoing the emotion 3304 of detest.Also, if
Biometric data 3302 indicates that the voice of individual is (for example, by using at least one microphone) to tremble, then can be true
The fixed individual is undergoing frightened emotion 3304.
Different biometric datas be can come from into different sources from the associated information such as different emotion.Example
Such as, described information can be based on laboratory result (for example, providing the correlation between different biometric datas and different conditions
The clinical data of (that is, existing or known correlation)), the two level knowledge (secondary of readme experiment and emotion
Knowledge) (for example, the use of individual in its communication for emoticon and/or word).Because some information are than other
Information is more reliable, so certain information can larger be weighted than other information.For example, in certain embodiments, clinical data
It is larger weighted than self-report data.In other embodiments, self-report data is larger weighted than clinical data.Laboratory (or learn
Practise) result may include from artificial neural network, C4.5, classification and/or regression tree, decision tree, deep learning, dimensionality reduction,
Elastomeric network, integrated study, expectation maximization, k- are average, k- arest neighbors, Density Estimator, core principle component analysis, linearly return
Return, logistic regression, matrix decomposition, naive Bayesian, neighbour's technology (neighbor techniques), Partial Least Squares Regression,
Random forest, ridge regression, support vector machines, multiple regression and/or it is all other be generally known to those skilled in the art
The data of habit technology.
Self-report data may include the data of their current state of individual identification, to allow to customize for the individual
(customize) biometric data.For example, computational linguistics can be not only used for what identification individual saying, but also can
With identification, how they say.In other words, the present invention can be used for analyzing and draw speech pattern (example associated with individual
Such as, allow present invention determine that who is speaking) and with the how associated speech pattern of subjects feel.For example, in response to " your today
How are you feeling today ", user may say " I am very happy now " or " I is hard hit now ".Computational linguistics can be used for according to a
The current affective state of body, mood, physical condition or the state of mind draw the difference in individual voice.Because the data are because of people
And it is different, so it is a kind of form of self-report data, and referred to herein as personalized artificial intelligence.When specific state is by certainly
When stating and detect and its state profile database is recorded by system, the self-report data of cross reference individual and other can be passed through
Biometric data (heart rate data etc.), it was demonstrated that (corroborated) and/or the accuracy for improving the data, the data
Be by analyze individual voice (and then by in system with the historical knowledge for the individual state that the time obtains and stores
Both databases in library and the potential wider other users state defined by the analysis to the voice of other users
Comparison) study about individual state data.
The gathered data for being substantially voice/mood profile of individual (is substantially the ID of the unique states profile of individual
Form) can by the system of collection biometric data using or with other systems (for example, individual smart phone, individual
Automobile, device (including Internet of Things (IOT) equipment or IOT system controls with voice or other biometric functions
Control equipment), internet or " cloud " storage device or any other calculating or machine with voice or other biometric functions
People's equipment (including but not limited to uses such as Apple's with the computer operating system with the individual ability interacted
The speech interface system of the Alexa of Siri, Google Assistant, Microsoft Cortana, Amazon and its follow-up system
The equipment of system operation) it is shared.Because shared information is unique for individual, and can be used for identifying the current shape of individual
State, so referred to herein as personalized artificial intelligence ID or " PAIID ".In one embodiment of the invention, readme number
According to be considered calibration data or can be used for verify, adjustment individual certain speech patterns or by individual certain
A little speech patterns and at least one state are (for example, at least one emotion, at least one mood, at least one physical condition or extremely
A kind of few state of mind) associated data.Data mark (labelling) and the deep learning side returned are utilized by application
Method and other technologies are apparent to those skilled in the art, by knowledge and predictability intrinsic in sustained improvement PAIID.
About computational linguistics, it should be understood that the present invention has surmounted identifies particular individual using simple speech analysis
Or what individual saying.But how the present invention can analyze individual acoustically (audibly) using computational linguistics
His/her is expressed to detect and determine at least one state, and is determined as using this and is provided a user content or hold
Element at least one action of row (for example, by action etc. of user's request).
It should be appreciated that the present invention is not limited to using single body or biological feature (for example, one group of biometric
Data) come determine individual state.Thus, for example it is surprised that eyeball expansion, facial expression and heart rate, which may be used to determine individual,
's.It is also understood that individual can once undergo more than one state, the biometric data received can be used to identify
More than one state, and system can help how most it determine using it to the analysis of the state or combinations of states of individual
Such as user's request or user instruction are responded well, or whether are done so really.It is also understood that the present invention is not limited to figure
The six kinds of emotions (that is, happy, angry, surprised, sad, detest and frightened) listed in 32, and can be used for identifying it is other (or
Substitute) affective state, such as cheerful and light-hearted, attraction, calmness, amusement, excited, tired, hungry, thirsty, health and happiness (well-being),
Failure, triumph, interest, enthusiasm, it is active, rouse oneself again, be satisfied with, regretting, liking, anxiety etc..Finally, the present invention is not limited to such as Figure 31
Shown in apply 3208, and can also include the one or more operated on web host 3102 and/or the network equipment 3106
Using.For example, the application operated on the network equipment 3106 or program can determine the feelings of individual using biometric data
Sense state, wherein affective state are transmitted to web host 3102 via WAN 3100.
Despite preferred embodiment, but the present invention is not limited to biometric data is used alone (for example, using sensing
Device, microphone and/or camera are collected) determine the current affective state or mood of individual.For example, the voice of individual is (individually
Or in combination with other biometric datas (blood pressure, the heart rate etc. of individual)) it is determined for the current body of individual
And/or mental health.The example of health include subjects feel how, it is such as healthy, good, poor, tired, exhausted, pain,
Pain and sick (including have a fever, have a headache, having a sore throat, its symptom of hyperemia etc.), the example of mental health includes the state of mind,
Such as brains is clear, fatigue, puzzlement, dizziness, lethargic sleep, without judgment and makes drunk.As an example, computational linguistics can be used for
Speech pattern is associated at least one body and/or the state of mind.This can use self-report data (for example, when individual sound
The voice of analysis individual whens claiming them to feel good, is not very comfortable, is puzzled etc.), this biometric data is linked to body
And/or the general data of the state of mind is (for example, by speech pattern (usually) and at least one body and/or state of mind phase
Associated data) or combinations thereof complete.Such system for example can be used for hospital determine the current body of patient and/or
The state of mind, and provide additional except the Standard physiological that is currently used in patient or hospital care or biometric label
Information.If making request or statement by patient or logical the problem of being generated to system responding, body and/or spiritual shape
State is above/below normal (N) (it may include the certain tolerance (T) in either direction) (for example, N+/- T), then can notify
Nurse or other medical workers.By the network analysis of the state to patient, this will be with the patient for such as providing extra level
Observation automation is provided about the early warning alarm of patient or the benefit of comfort.
As shown in figure 31, web host 3102 can also include other components, and user is such as allowed to key in the keyboard of data
3210, allow web host 3102 to user (or in the embodiment that biometric sensors are located inside web host 3102
To individual) display 3206 of display information, allow web host 3102 and external device communication (for example, via WAN 3100 with
The communication of the network equipment 3106 is communicated with the network equipment 3106 via wireless protocols, is passed via wireless protocols and external biometric
Sensor communication, etc.) transceiver 3212 and the internally and/or externally reception of the information of equipment and/or hair can be controlled
Send and/or run the processor 3202 using 3208 or relative machine readable instructions.
In one embodiment of the invention, the source (for example, content supplier) of the data based on web can indicate to
Individual under particular emotion state provides the interest of the data based on web.For example, as shown in figure 33, the content (example felt all right
Such as, in kitten of humour etc.) the owner can indicate to currently feeling that the individual of sad emotion provides should
The interest of content.Interest can simply be "Yes" or "No", or can be more complicated, such as 1-10 grades of interest.The present invention's
In another embodiment, the sources of the data based on web can indicate in response to something or other (for example, people, place, textual data
According to theme, the theme of video data, audio data theme etc.) and the individual that experienced particular emotion is provided based on web's
The interest of data.For example, as shown in figure 33, the owners of matchmaking services can indicate to when close to wedding (things) (for example,
The Email about wedding, viewing wedding video etc. are read in wedding church) when experienced happy emotion individual, they
Friend or they contact person provide relevant advertisements interest ($ 2.50CPM).As another example, the owner of jeweler's shop
It can indicate to when close to diamond (things) (for example, Email etc. in the shop, reading for selling diamond about diamond)
The individual that experienced excited emotion provides the interest (5.00CPC) of advertisement.Other data are also based on (for example, population is united
Count, profile data, the response etc. by click) selection content and/or interest based on web.Again, interest can be letter
Single "Yes" or "No", or can be more complicated, as the owner/source of 1-10 grades of interest, content is ready to show every time
The amount of money (CPM) of payment or the owner/source of content are ready to click the amount of money (CPC) of payment every time.
Another embodiment of the present invention can be related to and at least one auxiliary system (such as voice control or biometric peace
Total system) integrated system, the message wherein selected in emotion mainly alerts or safety recommendation, and is only specifically relevant feelings
Advertisement (will be described below) in condition.One example is the user that steering direction is received using speech recognition system,
Pulse and the voice data instruction of wherein user is angry.In this case, result can be customized (tailor) and existed by the present invention
Tranquil place nearby, it might even be possible to send out the mild warning of " more conventional for the pilot accident of excitement ".This, which is it, makes
Main purpose is not the example of emotion detection, but affection data can be collected from such system and by emotion number
According to for by message be aligned individual, contact person, caregiver, employer or even other subscription affective content data computers
System.Another alternative example is the security system that pulse and blood pressure are identified using retina scanning.If biometer
Measure that data are associated with sadness, then system can by go to communication equipment that it is connected rouse oneself or positive message come pair
Quasi- individual, or even alert care-giver.In other cases, it for example, for the vehicle equipped with automated driving system, is based on
To individual biometric feedback network analysis, in order to driver and other people (for example, the driver of passenger, other vehicles
Deng) interests, control loop can suggest operating or taking other actions with caution.
It should be noted that in the present invention, under certain service conditions, the agreement through user, individual private data is carried
Feed system, but in many cases, emotion response can be with time of day, place or given things (for example, jeweler's shop
Deng) associated, therefore individual identification information (PII) need not be shared with message provider.In example at a iewelry store, system letter
It singly is aligned with the individual of strong happy correlation and their friend.Although in some embodiments it is possible to individual
The chance for the PII that individual is shared with message provider is provided, but system can be without the information of this rank
Work.
Interesting data and other possible data can be used (for example, randomness, population are united using (Figure 31 is 3208)
Meter etc.) come it is determined that being supplied to the data (for example, advertisement etc.) based on web of individual.For example, if interesting data includes needle
Different bids to particular emotion or emotion-event relation, then application can (or network of relation be set to the individual of experience emotion
It is standby) advertisement associated with best bid is provided.In other embodiments, it is also examined when providing the data based on web to individual
Consider other data.In these embodiments, interesting data is considered when selection is supplied to the data based on web of individual
One standard.
It should be appreciated that having " response " of the individual of emotion response under particular state or to things, it is not limited to individual
Content based on web is provided, and may include any action with identified state consistency.In other words, identified
State can be used for determining background, referred herein as " scene background " by host (for example, automobile, smart phone etc.).For example, such as
Shown in Figure 39, automobile 4002 may include being determined (using the biometric data obtained via camera, microphone or sensor)
Driver's (not shown) is damaged or the host 4004 of changeable in mood (for example, anger, excitement etc.), can be switched to automatic Pilot,
Or the maximum speed of vehicle can be limited.In this embodiment, host can be based on the order that individual provides (for example, oral
Ground or other) and individual at least one emotion or mood execute " response ", wherein determining emotion/feelings based on biometric data
Thread.For example, can lead to robot in the voice command for executing action (by itself), with normal speed, (it can have battery
The benefit of protection, accuracy etc.) in the case of execution action, the voice command of the action is executed together with the life for indicating urgent mood
Object continuous data can cause robot to execute action at faster speed.
In one embodiment of the invention, host 4004 is the equipment for having network function, and be configured as via
Wide area network (WAN) 4000 is communicated at least one remote equipment (for example, 4006,4008,4010).For example, host 4004 can be with
It is configured as/being stored from remote data base (for example, " cloud ") 4010/and fetches individual state profile (for example, PAIID), and/
Or share individual state profile with other equipment (for example, 4006,4008) with network function.Profile can be stored for inciting somebody to action
Come fetch or shared profile so as to allow other equipment determine individual current state.As described above, host 4004 can be received
The feature of individual is linked to the self-report data of particular state by collection.By sharing the data with miscellaneous equipment, these equipment can be with
The current state for more easily determining individual collects readme (or calibration) data without (from individual).Database 4010 may be used also
Individual historic state or state (for example, history log of the original state of individual) whithin a period of time for storage.So
After can be used alone or be used in combination with daily record with other data, to determine individual during correlation time or period
(for example, when individual is increasing weight, when accident occurs, when execution noncontinuity or when specific action, etc.) state, or
Person determines the instruction about the psychological tendency or adaptability for executing certain functions, wherein such as individual state is heavy to closing
It wants, such as, but not limited to, the transaction instruction of drive an airplane, drive heavy goods vehicle or finance or commodity exchange house.
Status log can be further used for generation state " robot (bot) ", and being can be by of network distribution
The agency of body, to represent the information of individual searching as follows, the Info Link to individual has " interest " or wishes at them
The particular event being apprised of in a manner of positive or is passive under conditions of the particular state.
In alternative embodiments, the information of such as history log or individual state profile (for example, PAIID) etc also or
It is alternately stored on the memory devices 4024 in host 4004 (referring to Figure 40).In this embodiment, host 4004 can be with
Including transceiver 4032,4028 (referring to Figure 40) of processor 4022, display 4026 and at least one application, the work(of all of which
It can be identical as the function of similar assembly shown in Figure 31.Host 4004 can also include being configured as from individual (for example, vehicle
Driver) obtain audio/video at least one microphone and/or at least one camera 4030.As previously mentioned, audio/regard
Frequency is determined at least one state of individual.For example, can by individual voice and/or facial characteristics individually or with
Other data (for example, the heart rate data etc. obtained from the sensor on steering wheel) are analyzed in combination, to determine individual
At least one state.Then with use state at least one action can be executed.In one embodiment of the invention, state is used
In determining whether to execute the request (for example, order etc.) from individual, and if it is, whether should also execute other
Action (for example, limitation speed, offer warning etc.).For example, if driver's instruction vehicle launch of vehicle, vehicle (or its
In main operation) warning can be provided to driver in the case where determining driver fatigue, or can determine driver
Start automatic driving mode in the case of being impaired (for example, under the influence of this).In another example, aviation can be required
Company pilot provides the response for their how are you feeling todays, and how to be responded (by the property of its reply content according to pilot
Matter and its state of analysis), air traffic control can take the safety appropriate for acting and ensuring aircraft.In this case,
The case where can not providing any kind of response with similarity or background, will provide alarm, can indicate that pilot does not wish
In the case of hoping response (itself being information) or being not at response.
It should be appreciated that in the embodiment that individual responds things, things can be and individual close anything
Object, including people's (or equipment (such as smart phone etc.) of people), place (for example, based on GPS coordinate etc.) or shown to user
Content (for example, such as theme of the text data of Email, chat messages, text message or webpage, be included in it is for example electric
Sub- mail, chat messages, text message or webpage text data in word, the theme of video data, audio data master
Topic etc.)." things " or relative data can be supplied to web host by the network equipment, or can be web host
Known (for example, when individual responds the content based on web provided by web host, it can to its emotion response
To trigger additional data, such as advertisement).
The method according to an embodiment of the invention for executing the present invention is shown in FIG. 34.Since step 3500,
In step 3502, biometric data is received.As described above, biometric data can be individual at least one body and/
Or biological characteristic, including but not limited to heart rate, blood pressure, temperature, respiratory rate, facial characteristics, voice variation, eye movement and/or
The variation of expansion and chemical composition (in blood, sweat, saliva, urine or breathing).Then in step 3504, biometric data
For determining corresponding emotion, such as happy, angry, surprised, sad, detest or frightened.In step 3506, whether emotion is determined
It is the current state of individual, or whether is to be based on individual to things (for example, people, place, the information etc. shown to individual)
Response.If emotion be individual current state (step 3508), step 3512 based on individual current affective state come
Select the data based on web.However, if emotion is response (step 3510) of the individual to things, it is based in step 3512
Individual responds the emotion of things to select the data based on web.Then the data based on web of selection are carried in step 3514
Supply individual stops the processing in step 3516.
It should be appreciated that the present invention is not limited to methods shown in Figure 34, also, it include additional, less or different step
Rapid method is also within the spirit and scope of the present invention.For example, in step 3512, affection data (or emotion-things can be used
Data) and interesting data select the data based on web.As another example, in step 3514, can with (or substitute
Ground) to the third party (such as individual legal guardian, individual kinsfolk, medical personnel located (if individual in hospital
In), emergency response (if individual not within the hospital)) provide selected content (for example, data, text based on web disappear
Breath, Email etc.).Present invention is also not necessarily limited to be executed with any particular order described in Figure 34.For example, determining emotion
It is the response of the current state or individual of individual to things, can be executed before receiving biometric data.
Although biometric data etc. substantially can be very simple (for example, feature (such as blood that identification is measuring
Pressure) and measured value (such as 120/80)), but it can also be considerably complicated, to allow data to be stored for subsequently
It uses (for example, creating profile, chart etc.).For example, in one embodiment of the invention, as shown in figure 35, biometric passes
Sensor data may include detailed data, such as with reference to id (the technology unique identification of the basic point), entity id (user, team,
Local word or number (place word or number), equipment id), sensor mark (description measuring what character
Go here and there (string)), numerical value (integer or floating-point) and/or time (for example, GMT UNIX when a measurement is taken).Such as Figure 36 institutes
Show, emotion response data may include with reference to id (the technology unique identifier of the basic point), entity id (user, team, local word
Or number, equipment id), Emotion tagging (character string for being identified as emotion), time (the GMT UNIX when the record creates
Timestamp), emotional intensity (numerical value), and/or basic point (datum) create data (to what system creation basic point and/or which
Data are used to create the Technical Reference of the basic point).As shown in figure 37, emotion-item data may include with reference to id (basic point
Technology unique identifier), entity id (user, team, local word or number, equipment id), emotion is with reference to (to recording elsewhere
Particular emotion reference), things with reference to (reference to the specific matters recorded elsewhere), the time is (for example, create this note
GMT UNIX timestamps when record), correlation factor (indicate correlation rank numerical value, such as percentage), emotional intensity (number
Value), and/or basic point establishment data (the technology ginseng to what system creation basic point and/or which data for creating the basic point
It examines).As shown in figure 38, item data may include referring to id (the technology unique identifier of the basic point), entity id (user, group
Team, local word or number, equipment id), things with reference to (reference of specific " things " that is recorded elsewhere), the time (for example,
Create the GMT UNIX timestamps when record), correlation factor (indicate correlation rank numerical value, such as percentage) and/
Or basic point creates data (Technical Reference for being used to create the basic point to what system creation basic point and/or which data).It answers
Work as understanding, the present invention is not limited to serial datas shown in Figure 35-38, and transmit other methods of the data the present invention's
In spirit and scope.
The method of the execution present invention according to another embodiment of the invention is shown in FIG. 41.It is opened from step 4200
Begin, in step 4202, receives and ask from user.As described above, request may include the problem of user proposes (instruction response) or
Customer-furnished order (execution of instruction action).Then in step 4204, the request (or other biological stoichiometric number is analyzed
According to) to determine the current state of user, such as corresponding affective state, mood, physical condition and/or the state of mind.In step
4206, the current state of user is used to determine whether that specific action should be executed.For example, if the state of user is normal
, then requested action (for example, the action asked in step 4202) is executed in step 4210, terminates the party in step 4220
Method.If the state of user is abnormal, rather than troubling (for example, angry), then it can provide police in step 4212
It accuses.If the abnormal state of user and troubling (for example, drunk), in step 4208, can execute different dynamic
Make (for example, action different from the action asked in step 4202).If providing warning in step 4212, or in step
4208 perform different actions, then step 4220 and 4214 determine whether should to execute respectively requested action (for example,
In the action that step 4202 is asked).If answer is yes, requested action is executed in step 4210, in step 4220 knot
Beam this method.If answer is no, further action is not taken, terminates this method in step 4220.
It should be appreciated that the present invention is not limited to methods shown in Figure 41, also, it include additional, less or different step
Rapid method is within the spirit and scope of the present invention.Present invention is also not necessarily limited to the step described in Figure 41 is executed with any particular order
Suddenly.
The foregoing describe a kind of system and method, it is used to comment using at least readme, biometric and/or environmental data
The state (for example, mood etc.) for estimating user, to create databases from multiple such assessments of User Status so that system can be with
The state for more and more accurately analyzing user executes action (for example, requested dynamic using the User Status database of system
Work, alternative acts etc.), determine the user of (or prediction) under given background state (its can based on system to user with
The relevant biometric of self-report data of user and the cumulative knowledge of environmental data and be identified), and changed by self-report data
Determine that the ability of state of the user in specific background, self-report data may include that user determines (or prediction) system into system
State confirmation or correction.It is not intended to exhaustion or limits the invention to disclosed precise forms, also, in view of above-mentioned
Introduction, many modifications and variations are possible.It will be understood by those skilled in the art that there are various ways to realize preceding feature,
Also, the present invention is not limited to realize any ad hoc fashion of these features.The invention is solely defined by the appended claims.
Claims (20)
1. it is a kind of at least using self-report data and biometric data come determine user at least one state and in response to really
Fixed state includes the following steps come the method for executing at least one action：
Receive the first biometric data of the user by processor, first biometric data is from least one sensing
Device receives；
By the processor from the user receive the first self-report data, first self-report data with first biometer
The amount data essentially identical time receives and identifies the state of the user；
In memory by first biometric data and first self-report data storage, the first biometric number
According to being linked in relationship and in time first self-report data；
The second biometric data is received from the user by the processor, second biometric data is described first
It is received from least one sensor after self-report data；
First biometric data, second biometric data and described first are at least analyzed certainly by the processor
Data are stated, with the state of time of the determination user after having received second biometric data；And
At least one action is executed in response to identified state, the execution of at least one action is when described
Between before and one of the time at generation.
2. according to the method described in claim 1, wherein described first biometric data and second biometric data
Including pulse change, blood pressure change, heart rate variability, skin temperature, eyeball expansion, eye movement, facial muscles expression, breathing
At least one of rate, breathing chemical composition, sialochemistry ingredient, voice fundamental, sense of hearing change, cadence and body kinematics.
3. according to the method described in claim 1, the wherein described self-report data includes happy, sad, surprised, angry, detests, probably
Fear, is cheerful and light-hearted, is attraction, calmness, amusement, excited, tired, hungry, thirsty, health and happiness, failure, triumph, interest, enthusiasm, active, again
At least one of rouse oneself and be satisfied with.
4. according to the method described in claim 1, wherein described the step of receiving first self-report data from the user, wraps
Reception is included via Audiotex of the microphone from the user and via in haptic data of the input equipment from the user
It is at least one.
5. according to the method described in claim 1, further including the steps that receiving environmental data, the environment number by the processor
According at least one in the position including the user, the temperature at the position, height above sea level and time of day at the position
It is a, wherein the processor also using the environmental data determine the user the time state.
6. according to the method described in claim 1, wherein described by first biometric data and the first readme number
Further include the existing correlation stored between biometric data and different conditions, institute according to the step being stored in the memory
State processor also using at least part in the existing correlation determine the user the time state.
7. according to the method described in claim 1, wherein described first biometric data and first self-report data are together
It is stored in the single file in the memory, to which first biometric data is linked to described the in relationship
One self-report data.
8. according to the method described in claim 1, wherein at least one identification number is used for first biometric data
First self-report data is linked in relationship.
9. according to the method described in claim 1, wherein at least one of time started, timestamp, sample rate and offset are used
In first biometric data is linked to first self-report data in time.
10. according to the method described in claim 1, further comprising the steps of：At least analyze first biometric data with
Determine the user the time for receiving first biometric data state；And identified state is supplied to
The user, wherein first self-report data carries out following one：Confirm, deny and correct and is supplied to the user's
Identified state.
11. according to the method described in claim 1, further include the steps that from the user receive request to execute action, wherein
Described the step of executing at least one action in response to identified state be included in determine the state with it is requested
Requested action is executed under the execution unanimous circumstances of action, and is determining the execution of the state and requested action not
Alternative acts are executed under unanimous circumstances.
12. a kind of system for determining at least one state of user, including：
At least one processor；And
At least one processor equipment, and the processor communication, and for storing machine readable instruction fetch, the machine can
It reads instruction and is adapted for carrying out following steps：
The first biometric data of the user is received, first biometric data is received from least one sensor；
From the user receive the first self-report data, first self-report data with the basic phase of the first biometric data
The same time receives and identifies the state of the user；
First biometric data and first self-report data are stored in at least one processor equipment, institute
The first biometric data is stated in relationship and is linked to first self-report data in time；
The second biometric data is received from the user, second biometric data is having received first readme
It is received from least one sensor after data；
First biometric data, second biometric data and first self-report data are at least used, with true
The state of fixed time of the user after having received second biometric data；And
Execute at least one action in response to identified state, the execution of at least one action the time it
It is preceding and occur one of at the time, and include showing at least one visual aid for the user and be the user
Play at least one of at least one sound.
13. system according to claim 12, wherein first and second biometric data includes pulse change, blood
Press changes, heart rate variability, skin temperature, eyeball expansion, eye movement, facial muscles expression, respiratory rate, breathe it is chemical at
Point, sialochemistry ingredient, voice fundamental, sense of hearing change, at least one of cadence and body kinematics.
14. system according to claim 12, wherein the self-report data indicates that the user feels such as in specific time
What.
15. system according to claim 12, wherein the mobile device further includes for from described in user reception
At least one of microphone and keyboard of first self-report data.
16. system according to claim 12, wherein first biometric data and first self-report data one
It rises and is stored in the single file in at least one processor equipment, thus by first biometric data in relationship
On be linked to first self-report data.
17. in system according to claim 12, wherein timestamp, time started, offset, sample rate and hits extremely
Few one for being linked to first self-report data in time by first biometric data.
18. system according to claim 12, wherein machine-readable instruction is further adapted for receiving environmental data, wherein
The environmental data be additionally operable to determine the user the time state.
19. system according to claim 12, wherein machine-readable instruction is further adapted at least analyzing described first
State of the biometric data with the determination user in the time for receiving first biometric data, and by really
Fixed state is supplied to the user, wherein first self-report data carries out following one：Confirm and denies being supplied to described
The identified state of user.
20. a kind of system for determining the state of user, including：
At least one server is communicated with wide area network (WAN)；
Mobile device, via the WAN and at least one server communication, the mobile device includes：
At least one processor, for downloading machine-readable instruction from least one server；And
At least one processor equipment, for storing the machine readable instruction fetch, the machine readable instruction fetch is adapted for carrying out
Following steps：
The first biometric data of the user is received, first biometric data is received from least one sensor；
From the user receive the first self-report data, first self-report data with the basic phase of the first biometric data
The same time receives and identifies the state of the user；
First biometric data and first self-report data are stored in at least one processor equipment, institute
The first biometric data is stated in relationship and is linked to first self-report data in time；
The second biometric data is received from the user, second biometric data is having received first readme
It is received from least one sensor after data；
First biometric data, second biometric data and first self-report data are at least used, with true
The state of fixed time of the user after having received second biometric data；And
Execute at least one action in response to identified state, the execution of at least one action the time it
The preceding and generation one of at the time, and include at least one of the following：It shows on the mobile device at least
One image；At least one sound is played on the mobile device；The mobile device is set to vibrate；And via the WAN
At least one image is supplied to the second mobile device.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211121678.6A CN116389554A (en) | 2017-03-08 | 2017-03-08 | System for improving user's performance in athletic activities and method thereof |
| CN201710133899.8A CN108574701B (en) | 2017-03-08 | 2017-03-08 | System and method for determining user status |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710133899.8A CN108574701B (en) | 2017-03-08 | 2017-03-08 | System and method for determining user status |
Related Child Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211121678.6A Division CN116389554A (en) | 2017-03-08 | 2017-03-08 | System for improving user's performance in athletic activities and method thereof |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108574701A true CN108574701A (en) | 2018-09-25 |
| CN108574701B CN108574701B (en) | 2022-10-04 |
Family
ID=63576977
Family Applications (2)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710133899.8A Active CN108574701B (en) | 2017-03-08 | 2017-03-08 | System and method for determining user status |
| CN202211121678.6A Pending CN116389554A (en) | 2017-03-08 | 2017-03-08 | System for improving user's performance in athletic activities and method thereof |
Family Applications After (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211121678.6A Pending CN116389554A (en) | 2017-03-08 | 2017-03-08 | System for improving user's performance in athletic activities and method thereof |
Country Status (1)
| Country | Link |
|---|---|
| CN (2) | CN108574701B (en) |
Cited By (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109993093A (en) * | 2019-03-25 | 2019-07-09 | 山东大学 | Road anger monitoring method, system, equipment and medium based on face and respiratory characteristic |
| CN110598612A (en) * | 2019-08-30 | 2019-12-20 | 深圳智慧林网络科技有限公司 | Patient nursing method based on mobile terminal, mobile terminal and readable storage medium |
| CN111081020A (en) * | 2019-12-26 | 2020-04-28 | 安徽揣菲克科技有限公司 | Vehicle-mounted traffic accident early warning device based on cloud edge combination |
| CN111201744A (en) * | 2017-10-10 | 2020-05-26 | 查利·帕克斯 | System and method for providing alerts when transferring digital content |
| CN111564165A (en) * | 2020-04-27 | 2020-08-21 | 北京三快在线科技有限公司 | Data storage method, device, equipment and storage medium |
| CN112287397A (en) * | 2020-12-28 | 2021-01-29 | 曜立科技(北京)有限公司 | System and method for improving and guaranteeing safety of patient information transmission |
| WO2021033104A1 (en) * | 2019-08-22 | 2021-02-25 | International Business Machines Corporation | Adapting movie storylines |
| CN112954708A (en) * | 2021-03-22 | 2021-06-11 | 中电积至(海南)信息技术有限公司 | Positioning method and device based on measurement report |
| CN112949403A (en) * | 2021-02-01 | 2021-06-11 | 浙江大学 | Reliable user authentication method and system based on biological characteristics of mandible |
| CN113257424A (en) * | 2021-07-16 | 2021-08-13 | 成都厚信健科技有限公司 | Telemedicine system based on big data |
| WO2022037567A1 (en) * | 2020-08-18 | 2022-02-24 | International Business Machines Corporation | Biometric user authentication |
| CN114379563A (en) * | 2020-10-21 | 2022-04-22 | 广州汽车集团股份有限公司 | Safe driving control method and system and storage medium |
| EP3891691A4 (en) * | 2018-12-07 | 2022-11-09 | Warner Bros. Entertainment Inc. | Trip-configurable content |
| CN117438091A (en) * | 2023-12-20 | 2024-01-23 | 山西工程科技职业大学 | Motion intensity detection system and method based on big data |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101896926A (en) * | 2007-10-31 | 2010-11-24 | 安东尼·理查德·罗思柴尔德 | System and method for adding an advertisement to a personal communication |
| CN102933136A (en) * | 2010-06-07 | 2013-02-13 | 阿弗科迪瓦公司 | Mental state analysis using web services |
| CN103959756A (en) * | 2011-11-29 | 2014-07-30 | 索尼爱立信移动通讯有限公司 | Portable electronic equipment and method of recording media using a portable electronic equipment |
| CN105830066A (en) * | 2013-12-19 | 2016-08-03 | 微软技术许可有限责任公司 | Tagging images with emotional state information |
| US20160241533A1 (en) * | 2011-11-07 | 2016-08-18 | Anurag Bist | System and Method for Granular Tagging and Searching Multimedia Content Based on User's Reaction |
-
2017
- 2017-03-08 CN CN201710133899.8A patent/CN108574701B/en active Active
- 2017-03-08 CN CN202211121678.6A patent/CN116389554A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101896926A (en) * | 2007-10-31 | 2010-11-24 | 安东尼·理查德·罗思柴尔德 | System and method for adding an advertisement to a personal communication |
| CN102933136A (en) * | 2010-06-07 | 2013-02-13 | 阿弗科迪瓦公司 | Mental state analysis using web services |
| US20160241533A1 (en) * | 2011-11-07 | 2016-08-18 | Anurag Bist | System and Method for Granular Tagging and Searching Multimedia Content Based on User's Reaction |
| CN103959756A (en) * | 2011-11-29 | 2014-07-30 | 索尼爱立信移动通讯有限公司 | Portable electronic equipment and method of recording media using a portable electronic equipment |
| CN105830066A (en) * | 2013-12-19 | 2016-08-03 | 微软技术许可有限责任公司 | Tagging images with emotional state information |
Cited By (25)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111201744A (en) * | 2017-10-10 | 2020-05-26 | 查利·帕克斯 | System and method for providing alerts when transferring digital content |
| EP3891691A4 (en) * | 2018-12-07 | 2022-11-09 | Warner Bros. Entertainment Inc. | Trip-configurable content |
| CN109993093A (en) * | 2019-03-25 | 2019-07-09 | 山东大学 | Road anger monitoring method, system, equipment and medium based on face and respiratory characteristic |
| GB2602904B (en) * | 2019-08-22 | 2024-03-13 | Ibm | Adapting movie storylines |
| WO2021033104A1 (en) * | 2019-08-22 | 2021-02-25 | International Business Machines Corporation | Adapting movie storylines |
| US11429839B2 (en) | 2019-08-22 | 2022-08-30 | International Business Machines Corporation | Adapting movie storylines |
| GB2602904A (en) * | 2019-08-22 | 2022-07-20 | Ibm | Adapting movie storylines |
| CN110598612A (en) * | 2019-08-30 | 2019-12-20 | 深圳智慧林网络科技有限公司 | Patient nursing method based on mobile terminal, mobile terminal and readable storage medium |
| CN110598612B (en) * | 2019-08-30 | 2023-06-09 | 深圳智慧林网络科技有限公司 | Patient nursing method based on mobile terminal, mobile terminal and readable storage medium |
| CN111081020A (en) * | 2019-12-26 | 2020-04-28 | 安徽揣菲克科技有限公司 | Vehicle-mounted traffic accident early warning device based on cloud edge combination |
| CN111564165A (en) * | 2020-04-27 | 2020-08-21 | 北京三快在线科技有限公司 | Data storage method, device, equipment and storage medium |
| CN111564165B (en) * | 2020-04-27 | 2021-09-28 | 北京三快在线科技有限公司 | Data storage method, device, equipment and storage medium |
| WO2022037567A1 (en) * | 2020-08-18 | 2022-02-24 | International Business Machines Corporation | Biometric user authentication |
| US11604862B2 (en) | 2020-08-18 | 2023-03-14 | International Business Machines Corporation | Biometric user authentication |
| GB2613304A (en) * | 2020-08-18 | 2023-05-31 | Ibm | Biometric user authentication |
| CN114379563A (en) * | 2020-10-21 | 2022-04-22 | 广州汽车集团股份有限公司 | Safe driving control method and system and storage medium |
| CN114379563B (en) * | 2020-10-21 | 2023-08-08 | 广州汽车集团股份有限公司 | Safe driving control method, system and storage medium thereof |
| CN112287397B (en) * | 2020-12-28 | 2021-04-23 | 曜立科技(北京)有限公司 | System and method for improving and guaranteeing safety of patient information transmission |
| CN112287397A (en) * | 2020-12-28 | 2021-01-29 | 曜立科技(北京)有限公司 | System and method for improving and guaranteeing safety of patient information transmission |
| CN112949403B (en) * | 2021-02-01 | 2022-08-23 | 浙江大学 | Reliable user authentication method and system based on biological characteristics of mandible |
| CN112949403A (en) * | 2021-02-01 | 2021-06-11 | 浙江大学 | Reliable user authentication method and system based on biological characteristics of mandible |
| CN112954708A (en) * | 2021-03-22 | 2021-06-11 | 中电积至(海南)信息技术有限公司 | Positioning method and device based on measurement report |
| CN113257424A (en) * | 2021-07-16 | 2021-08-13 | 成都厚信健科技有限公司 | Telemedicine system based on big data |
| CN117438091A (en) * | 2023-12-20 | 2024-01-23 | 山西工程科技职业大学 | Motion intensity detection system and method based on big data |
| CN117438091B (en) * | 2023-12-20 | 2024-03-01 | 山西工程科技职业大学 | Motion intensity detection system and method based on big data |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116389554A (en) | 2023-07-04 |
| CN108574701B (en) | 2022-10-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20210005224A1 (en) | System and Method for Determining a State of a User | |
| CN108574701A (en) | System and method for determining User Status | |
| US20200381021A1 (en) | System and Method for Improving User Performance in a Sporting Activity | |
| US10901509B2 (en) | Wearable computing apparatus and method | |
| US10261947B2 (en) | Determining a cause of inaccuracy in predicted affective response | |
| US9955902B2 (en) | Notifying a user about a cause of emotional imbalance | |
| CN103561652B (en) | Method and system for assisting patients | |
| US8446275B2 (en) | General health and wellness management method and apparatus for a wellness application using data from a data-capable band | |
| US20160055236A1 (en) | Personalized experience scores based on measurements of affective response | |
| US11024339B2 (en) | System and method for testing for COVID-19 | |
| US20140122102A1 (en) | General health and wellness management method and apparatus for a wellness application using data associated with data-capable band | |
| US20140129243A1 (en) | General health and wellness management method and apparatus for a wellness application using data associated with a data-capable band | |
| US20140129007A1 (en) | General health and wellness management method and apparatus for a wellness application using data associated with a data-capable band | |
| US20190108191A1 (en) | Affective response-based recommendation of a repeated experience | |
| CA2814684A1 (en) | Activity attainment method and apparatus for a wellness application using data from a data-capable band | |
| Chung et al. | Real‐world multimodal lifelog dataset for human behavior study | |
| US20140129008A1 (en) | General health and wellness management method and apparatus for a wellness application using data associated with a data-capable band | |
| US20140129242A1 (en) | General health and wellness management method and apparatus for a wellness application using data associated with a data-capable band | |
| US20140127650A1 (en) | General health and wellness management method and apparatus for a wellness application using data associated with a data-capable band | |
| US20140129239A1 (en) | General health and wellness management method and apparatus for a wellness application using data associated with a data-capable band | |
| US20140125493A1 (en) | General health and wellness management method and apparatus for a wellness application using data associated with a data-capable band | |
| CA2795978A1 (en) | General health and wellness management method and apparatus for a wellness application using data associated with a data-capable band | |
| US20140127649A1 (en) | General health and wellness management method and apparatus for a wellness application using data associated with a data-capable band | |
| US20140125480A1 (en) | General health and wellness management method and apparatus for a wellness application using data associated with a data-capable band | |
| US20140125481A1 (en) | General health and wellness management method and apparatus for a wellness application using data associated with a data-capable band |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| REG | Reference to a national code |
Ref country code: HK Ref legal event code: DE Ref document number: 1260812 Country of ref document: HK |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |