CN105007525A - Interactive situation event correlation smart perception method based on application of smart television - Google Patents
Interactive situation event correlation smart perception method based on application of smart television Download PDFInfo
- Publication number
- CN105007525A CN105007525A CN201510312111.0A CN201510312111A CN105007525A CN 105007525 A CN105007525 A CN 105007525A CN 201510312111 A CN201510312111 A CN 201510312111A CN 105007525 A CN105007525 A CN 105007525A
- Authority
- CN
- China
- Prior art keywords
- intelligent television
- behavior
- user
- staff
- state
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/41—Structure of client; Structure of client peripherals
- H04N21/422—Input-only peripherals, i.e. input devices connected to specially adapted client devices, e.g. global positioning system [GPS]
- H04N21/42204—User interfaces specially adapted for controlling a client device through a remote control device; Remote control devices therefor
- H04N21/42206—User interfaces specially adapted for controlling a client device through a remote control device; Remote control devices therefor characterized by hardware details
- H04N21/42222—Additional components integrated in the remote control device, e.g. timer, speaker, sensors for detecting position, direction or movement of the remote control, microphone or battery charging device
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/41—Structure of client; Structure of client peripherals
- H04N21/422—Input-only peripherals, i.e. input devices connected to specially adapted client devices, e.g. global positioning system [GPS]
- H04N21/42204—User interfaces specially adapted for controlling a client device through a remote control device; Remote control devices therefor
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/442—Monitoring of processes or resources, e.g. detecting the failure of a recording device, monitoring the downstream bandwidth, the number of times a movie has been viewed, the storage space available from the internal hard disk
- H04N21/44213—Monitoring of end-user related data
- H04N21/44218—Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV program
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/47—End-user applications
- H04N21/485—End-user interface for client configuration
- H04N21/4852—End-user interface for client configuration for modifying audio parameters, e.g. switching between mono and stereo
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/47—End-user applications
- H04N21/485—End-user interface for client configuration
- H04N21/4858—End-user interface for client configuration for modifying screen layout parameters, e.g. fonts, size of the windows
Abstract
The invention provides an interactive situation event correlation smart perception method based on applications of a smart television, and belongs to the field of smart televisions. The method includes the steps: capturing context information tag of an interactive situation event state at a current K moment and also an effective continuous historical frame image (frames) of a user posture behavior track before the K moment; identifying the posture behavior of a user at the current moment by matching the user posture behavior track with a basic body behavior in a posture behavior model database, and then identifying an interactive situation event occurring at the current moment; and speculating an interactive situation event that will activate and occur in the future on the basis of the identified interactive situation event.
Description
Technical field
The invention belongs to intelligent television field, be specifically related to a kind of exchange scenario event correlation Intellisense method towards intelligent television application.
Background technology
The development of modern intelligent computation and the communication technology, while strengthening TV functions and interactive performance, is also that the development of man-machine interaction brings new challenge.In the epoch that intelligent television is popularized rapidly, user how quickness and high efficiency, naturally become a key issue alternately with intelligent television easily.In recent years, the user figure behavior interaction technique being representative with human body natural's gesture is applied in fields such as Sign Language Recognition, windows order control, intelligent television manipulations, and nature, high efficiency, accessible, intelligentized human-computer interaction interface have become the Main way of New Generation of Intelligent man-machine interaction development.
Intelligent television more and more becomes the interactive object that people carry out amusement, study, even indoor exercise, music, somatic sensation television game, viewing film and TV programme or even online social and media content of networking.By the restriction of traditional tv remote controller, user cannot enrich more with intelligent television, naturally manipulate mutual.With the development of human-computer interaction technology, NaserH and Mothammad utilizes Gesture Recognition to produce control command, realizes between user and computer mutual by these control commands.The Hi-Touch that Hisense's intelligent television is released, can pass through motion recognition system, realize interaction and the somatic sensation television game of user and intelligent television.Samsung adopts the operation detection system of cmos imageing sensor in intelligent television, can be detected the hand motion of operator, can carry out multiple simple motion action by camera.The gesture recognition system of view-based access control model obtains application in intelligent television, compensate for TV remote controller uses intelligent television function interactive mode defect to user.
But due to complicated variety, the factor of natural environment impact of the behavior of figure's gesture, the in actual applications main problem that there are following three aspects:
One is that compound action Activity recognition rate is too low;
Two is that the operational motion behavior complexity that user needs to remember is various, and cognitive load is larger;
Three is feel that difficulty and long posture action operation can allow user feel tired to completing compound action behavior in user operation process, and operational load is larger.
The people such as Wei-Po Lee (please refer to Wei-Po Lee, Che Kaoli, Jhih-Yuan Huang.A smart TV system with body-gesture control, tag-based rating and context-aware recommendation [J] .Knowledge-Based Systems, 2013) kinect body sense video camera is utilized to achieve the interactive mode of nature gesture control intelligent television, create a kind of commending system based on the situation contextual information residing for social activity mark and user, for user recommends the service content of the most applicable users ' individualized requirement.Thisly incorporate the situation contextual information commending contents service that user uses intelligent television, alleviated cognition and the operation burden of user to a certain extent, but do not consider that figure's behavior contextual information of user itself is on the impact alleviating user interactions.
Summary of the invention
The object of the invention is to solve the difficult problem existed in above-mentioned prior art, provide a kind of exchange scenario event correlation Intellisense method towards intelligent television application, effective perception user view, meets natural, harmonious, the intelligent requirement of man-machine interaction.
The present invention is achieved by the following technical solutions:
A kind of exchange scenario event correlation Intellisense method, catches the historical frames image frames effectively continuously of current K moment exchange scenario state-event contextual information tag and the user figure action trail before the K moment;
Mate with the basic figure's behavior in figure's behavior model pattern database, identify figure's behavior that user is current, pick out the exchange scenario event of current generation accordingly;
Infer according to the exchange scenario event picked out and the exchange scenario event that future time instance will activate generation.
Described method utilizes kinect3D body sense equipment to obtain the historical frames image frames effectively continuously of the user figure action trail before the K moment.
Described basic figure's behavior comprises:
WF: wave forward
WA: wave in left and right
WUD: wave up and down
Fist: clench fist.
Described method comprises:
S1, initialization exchange scenario event correlation relation drives figure;
S2. detect the dynamic action information of human body gesture in the intelligent television broadcast state in K moment and the successive image frame of kinect3D body sense equipment acquisition, catch figure's behavior;
S3, mates with human body behavior model basic in human body behavioral data pattern base, then returns S2.
Described exchange scenario event correlation relation drives figure as follows:
EDG＝<(WPU‐SmartTV,cxt).E,(WPU‐SmartTV,cxt).V>；
Wherein, WPU ?SmartTV represent that user uses intelligent television to watch the scene of TV programme, cxt represents situational contexts information;
(WPU‐SmartTV,cxt).V＝(V
0,V
1,V
2,V
3,V
4)；
Wherein, V
0program broadcast state sight event, V
1that body sense gesture operation function opens sight event, V
2channel adjustment sight event, V
3volume adjusting sight event, V
4that gesture operation function closes sight event;
(WPU‐SmartTV,cxt).E＝(e
01,e
12,e
13,e
22,e
23,e
24,e
32,e
33,e
34,e
41)
Wherein, e
01for WF; e
12, e
32for WUD; e
13, e
23for WA; e
24, e
34for Fist.
Described S1 is achieved in that
A1, obtain the basic function status indication parameter of intelligent television, be input in the array of vertex set (WPU ?SmartTV, cxt) .V;
A2, the basic figure's behavior in basic figure's behavior model database is stored in the array of limit collection (WPU ?SmartTV, cxt) .E;
A3, arrange the initial effective status value tag=0 of EDG, intelligent television initial condition is V
0.
Described S2 is achieved in that
B1, obtain the basic function status indication parameter of intelligent television, and assignment is to tag; Tag=0 represents program broadcast state; Tag=1 represents and opens body sense operating function state; Tag=2 represents channel mode of operation; Tag=3 represents volume operation state; Tag=4 represents closure body sense operating function;
B2, as tag=1, obtain the action contextual information of human body gesture: utilize Kinect 3D body sense equipment to obtain human depth's information, obtain the dynamic gesture motion track information of staff, the deep image information that recycling Kinect3D body sense equipment gets and RGB image information, utilize nature staff complexion model to Image Segmentation Using, obtain the static gesture image of nature staff.
Described B2 comprises:
B21, unlatching Kinect3D body sense equipment, follow the tracks of staff, obtain the center-of-mass coordinate of staff;
In B22, every continuous print M two field picture, calculate the mobile Euclidean distance mean value s1 of staff barycenter in M two field picture; M>20; If s1<50 millimeter, then from the movement locus of center of mass point and the static gesture of staff of M+1 image frame grabber staff;
B23, in the staff gesture contextual information gatherer process from M+1 two field picture, if from N two field picture, the mean value s2 of the Euclidean distance of staff center-of-mass coordinate movement in the image of continuous 20 frames, if s2<50 millimeter, then from N two field picture, stop gathering the movement locus of staff center of mass point and the information gathering of staff static gesture; N>M+30;
B24, from M+1 two field picture to N two field picture the staff center-of-mass coordinate point movement locus that collects and staff static gesture be the dynamic action information of the human body gesture collected.
Carry out mating being achieved in that with human body behavior model basic in human body behavioral data pattern base in described S3
(1) V is in current intelligent television state
0time, if the figure's behavior now capturing user is mated with WF, then trigger V
1generation; Otherwise intelligent television will remain on and activate V
0state;
(2) V is at current intelligent television
1during state of activation, if the figure's behavior under current context is mated with WA, then trigger V
2generation, if figure's behavior is mated with WUD, then trigger V
3generation; Otherwise intelligent television will remain on and activate V
1state;
(3) V is in intelligent television state
2time, if the figure's behavior now capturing user is mated with WA, then trigger V
3generation, if the figure's behavior now capturing user is mated with Fist, then trigger V
4generation; Otherwise intelligent television will remain on and activate V
2state;
(4) V is at current intelligent television
3time, if the figure's behavior now capturing user is mated with WA, then trigger V
2generation, if the figure's behavior now capturing user is mated with Fist, then trigger V
4generation; Otherwise intelligent television will remain on and activate V
3state;
(5) V is at current intelligent television
4time, perform closure body sense operating function if completed, then automatically jump to and activate V
0state, now label tag value is set to 0.
Compared with prior art, the invention has the beneficial effects as follows: the present invention effectively can realize the object of perception user view, meet the requirement of the natural, harmonious, intelligent of man-machine interaction and fluency.
Accompanying drawing explanation
Fig. 1 exchange scenario event correlation figure
Fig. 2 Kinect space coordinates schematic diagram
The dynamic gesture movement locus characteristic image got in Fig. 3 a embodiment
The static gesture image got in Fig. 3 b embodiment
The step block diagram of Fig. 4 the inventive method
Fig. 5 experiment porch module rack of the present invention composition.
Fig. 6 is the present invention four kinds of user's basic figure's operational motion behavior associated description figure.
Embodiment
Below in conjunction with accompanying drawing, the present invention is described in further detail:
For the problem that the user operation load and cognitive load that solve the existence in intelligent television manipulation of body sense gesture recognition system are larger, the present invention proposes a kind of exchange scenario event perception correlation model towards intelligent television application, utilize contextual status contextual information and user figure behavior contextual information, identification exchange scenario perception events, make the intention under the computing equipment perception user situation of presence, (Xu Guang Yu is please refer to by a kind of implicit interactions, Tao Linmi, Shi Yuanchun, Zhang Xiang. the man-machine interaction [J] under common calculating model. Chinese journal of computers, 2007, mode 07:1041-1053), the operation object of the completing user of implicit expression.For this special scenes example of WPU-SmartTV, based on the Intellisense control system of Kinect 3D body sense operative intelligence TV, structure user commonly uses, natural figure's interbehavior model of simple and flexible, gesture motion track before Real-time Collection user current time in N frame consecutive image and static gesture change, according to the basic figure's behavior model in figure's behavior model pattern database, gesture motion track before identifying user's current time in N frame consecutive image and static gesture change (please refer in quiet, field Congress, Yin Jianqin. based on the dynamic hand gesture recognition algorithm [J] of depth information. journal of Shandong university (engineering version), 2014, 03:52-56+63. and Yang X W, Feng Z Q, Huang Z Z, et al.A Gesture Recognition Algorithm Using Hausdorff-Like Distance Template Matching Based on the Main Direction of Gesture [C] //Applied Mechanics and Materials.2015, 713:2156-2159), user figure behavior contextual information under generation current state.And take out the basic exchange scenario event existed in WPU-SmartTV, according to basic natural figure's behavior model and basic exchange scenario state-event, and the incidence relation existed between basic exchange scenario perception events, the intention that perception user is current, realizes natural, harmonious, the intelligentized man-machine interaction of user and intelligent television.
In the gesture interaction system of user and interactive intelligence TV, as mutual single input channel mode, there is user cognition load and the larger problem of operational load in gesture motion behavior.For above-mentioned Problems existing, propose the contextual formal definitions of situation---exchange scenario event in user and intelligent television reciprocal process, establish exchange scenario event correlation sensor model.This model uses intelligent television to watch TV programme (watch program use smart TV for user, WPU-SmartTV) this special scenes, take out the basic exchange scenario event wherein existed, analyze the contextual status context of the current time of basic exchange scenario event and user figure behavior (in the present invention, gesture motion behavior is refered in particular in user figure behavior), the contingent exchange scenario event of future time instance is inferred according to incidence relation possible between basic exchange scenario event, the intention of perception user, realize the alternately intelligentized of user and intelligent television.In conjunction with WPU-SmartTV example, give the implementation algorithm of exchange scenario event correlation sensor model.Verify by experiment, this model can be intended to according to the operation of situation context Accurate Prediction user, realizes user and the mutual harmony of intelligent television, nature and intelligent.
Exchange scenario event correlation sensor model of the present invention is as follows:
Exchange scenario event perception towards intelligent television focuses on extracting user and the mutual elemental motion behavior of intelligent television, again in conjunction with the feature operation state residing for current time intelligent television, the exchange scenario event of identification intelligent television current time, utilize the imminent exchange scenario event of figure inference logic model presumes subsequent time, and then the operation intention of perception user current time.According to user using the interaction analysis with intelligent television in intelligent television process, the realization of sight event correlation sensor model is divided into 3 steps: the 1. foundation of figure's behavior model database; 2. figure inference logic model; 3. user view perception algorithm.
Concrete steps are as follows:
The foundation of figure's behavior model database:
In the present invention, human hands dynamic behaviour or static gesture Behavioral change are refered in particular in user figure behavior.The foundation of figure's behavior model database, by research user experience,, the most natural basic operation behavior model that the most often use according to user in user experience behavior and mentality discovery reciprocal process, to the basic act model Modling model database extracted, and add in pattern database.
By using intelligent television to watch analysis with intelligent television interbehavior in TV programme process to a large amount of Different age group intelligent television user, be extracted wherein four kinds of users often use and the most naturally, basic figure's behavior model flexibly.The behavior of four kinds of user's basic figure's operational motions [The basic body behavior-referred to as BBB] associated description is as shown in Figure 6.
First need to extract the human body basic act state in user and intelligent television reciprocal process (to please refer to Fan Yinting, Teng Dongxing, Yang Haiyan, Ma Cuixia, Dai Guozhong, Wang Hongan. based on the adaptive user interface model [J] of experience perception. Chinese journal of computers, 2011,11:2211 ?2223).After determining four kinds of basic figure's operation behavior models, behavior model training is carried out to all ages and classes, the user in sex stage, generation model database, and add basic figure's operation behavior model of Four types to pattern database.
Figure inference logic model is as follows:
After system acquisition to the posture action behavior of user under current state, in conjunction with the feature operation state residing for current time intelligent television, the i.e. occurent sight event of intelligent television this moment, infers the imminent exchange scenario event of subsequent time by certain logic rules.User operation demand for services in the present invention based on exchange scenario event context information is known, and user interactions service discovery and content of operation are unknown, system needs to carry out monitoring in real time to user figure behavior and judges user behavior state, because the uncertainty of user figure behavior, intelligent television state may change at any time, these uncertain factors bring great cognitive load (user needs a lot of gesture operation order remembered) to user operation intelligent television and (sense tired out of user's long-time operation) is born in operation.
For this reason, the present invention proposes a kind of figure inference logic model based on exchange scenario event context information, be that a directed graph (please refer to not same by basic sight event correlation contextual definition, Chu Weijie, Li Weiping, Wu Zhonghai, Lin Huiping. based on the active service discovery method of context aware event. Journal of Software, 2011, 22 (Suppl. (2)): 41 ?51), according to the situation of presence event perceived, composition graphs reasoning from logic model (please refer to a flower bud. represent rule and inference method [J] thereof with concept map. Northwest University's journal (natural science edition), 1994, 04:319 ?322), infer the current and following next exchange scenario event that may activate of user.
Figure inference logic model of the present invention is as follows:
The exchange scenario event set incidence relation that situation contextual information cxt is relevant in special scenes sc can be defined as directed graph EDG=< (sc, a cxt) .V, (sc, cxt) .E>;
(1) vertex set V={V
1, V
2..., V
nbe context aware event set, represent the contingent value change with business implication of exchange scenario event context information cxt in scene sc.V
i: represent an exchange scenario perception events in scene sc.
(2) limit collection (sc, cxt) .E={e
11, e
12..., e
mnbe event-driven incidence set, represent the value change with business implication that current human figure behavior contextual information sc occurs in scene sc.E
ij: represent at context aware event V
iunder the condition occurred, the change of human body figure actuating signal contextual information cxt is by triggered scenario perception events V
jgeneration.
(3) V
i---e
ij---V
j: represent that current scene is in exchange scenario event V
iduring state, e
ijgeneration drive exchange scenario event V
jgeneration.
(4) exchange scenario event situation context (cxt) comprises two parts: user figure behavior contextual information (User Posture Behavior Context, UPBC), exchange scenario state-event contextual information (Status of Environment Context, SEC); Wherein, cxt=(UPBC, SEC).
The definition of basic exchange scenario event is: in interactive process at a time or the time period in keep certain specific function state, it is a basic exchange scenario event that so this functional status is just defined as.
In the present invention, intelligent television is used to watch TV programme (watch program use smart TV to user, WPU ?SmartTV) this special scenes of interactive process of carrying out basic operation analyzes, generate exchange scenario event correlation relation drive figure EDG=< (WPU ?SmartTV, cxt) .E, (WPU ?SmartTV, cxt) .V>.
I, take out WPU ?5 basic exchange scenario events in SmartTV scene:
V respectively
0: program broadcast state sight event, V
1: body sense gesture operation function opens sight event, V
2: channel adjustment sight event, V
3: volume adjusting sight event, V
4: gesture operation function closes sight event.
(WPU‐SmartTV,cxt).V＝(V
0,V
1,V
2,V
3,V
4)。
II, exchange scenario event correlation driving figure limit collection have following corresponding relation with the figure's behavior model activated corresponding to exchange scenario event
e
01：WF；e
12、e
32：WUD；e
13、e
23：WA；e
24、e
34：Fist
(WPU‐SmartTV,cxt).E＝(e
01,e
12,e
13,e
22,e
23,e
24,e
32,e
33,e
34,e
41)。
According to above information, build and generate figure inference logic model directed graph EDG=< (sc, cxt) .V, (sc, cxt) .E>, and with the ergodic process of directed graph for reasoning from logic judgement is carried out in direction, figure inference logic model as shown in Figure 1.
As shown in Figure 4, user view perception algorithm (intention of user perception algorithm, IPA) is as follows:
The realization of user view perception, first the state context information of perception current exchange scenario event and the current figure's behavior contextual information of user is needed, mate with the basic figure's behavior in figure's behavior model pattern database, comprise the action of staff and the static change of gesture, identify figure's behavior that user is current, the exchange scenario event of the current generation of identification accordingly, according to the exchange scenario event picked out utilize figure inference logic model presumes to go out exchange scenario event that future time instance will activate generation, with this realize perception operation intention of user object (please refer to Gu Junzhong. context aware calculate [J]. East China Normal University's journal (natural science edition), 2009, 05:1-20+145.).
Algorithm realization process is as follows:
Input: in WPU-SmartTV scene, the historical frames image frames effectively continuously of the user figure action trail before the K moment that current K moment exchange scenario state-event contextual information tag, kinect3D body sense equipment (being also called body sense video camera) captured obtains;
The exchange scenario event V of Output:K+1 time trigger
k+1and the operation intention of user;
Step is as follows:
Step1. initialization exchange scenario event correlation relation drives figure EDG=< (WPU ?SmartTV, cxt) .E, (WPU ?SmartTV, cxt) .V>.
Initialization procedure is as follows:
1, obtain the basic function status indication parameter of intelligent television, be input in the array of vertex set (WPU ?SmartTV, cxt) .V of storage figure inference logic;
2, four kinds of basic figure's behavioral data parameters in four of Fig. 6 kinds of basic figure's behavior model databases are read in figure's Activity recognition module of system, be stored in e in the array of (WPU ?SmartTV, cxt) .E
01: WF; e
12, e
32: WUD; e
13, e
23: WA; e
24, e
34: Fist.1,2 liang of steps complete the initialization to reasoning from logic figure EDG.After initialization completes, the initial effective status value arranged in reasoning from logic figure is, tag=0, intelligent television initial condition is V
0;
Step2. detect the dynamic action information of human body gesture in the intelligent television broadcast state tag in k moment and the successive image frame of kinect3D body sense equipment acquisition, catch figure's behavior [posture behavior model, referred to as PBM], specific as follows:
Obtain the basic function status indication parameter of intelligent television, and assignment is to tag:tag=0, represents program broadcast state; Tag=1, represents and opens body sense operating function state; Tag=2, represents channel mode of operation; Tag=3, represents volume operation state; Tag=4, represents closure body sense operating function.
Obtain the action contextual information process of human body gesture: utilize Kinect 3D body sense equipment, obtain human depth's information, configuration OpenNI (a kind of natural interaction architecture platform of increasing income) realizes the tracking to staff by the handsGenerator in OpenNI, obtain the dynamic gesture motion track information of staff, the deep image information that recycling Kinect gets and RGB image information, utilize nature staff complexion model to Image Segmentation Using, obtain the static gesture image of nature staff.
The dynamic gesture motion track information obtained by Kinect3D body sense video camera and OpenNI natural interaction platform framework and static gesture image information are the action contextual information of human body gesture.
Detailed process:
1, open Kinect camera, follow the tracks of staff, obtain the center-of-mass coordinate of staff;
2, in every continuous print M (M>20) two field picture, the mobile Euclidean distance mean value s1 of staff barycenter in M two field picture is calculated:
On two dimensional surface, 2 a (x1, y1) are as follows with the Euclidean distance between b (x2, y2):
If s1<50 millimeter, then from the movement locus (as shown in Figure 3 a) of the center of mass point of M+1 image frame grabber staff and the static gesture (as shown in Figure 3 b) of staff;
3, in the staff gesture contextual information gatherer process from M+1 two field picture, if from N (N>M+30) two field picture, the mean value s2 of the Euclidean distance of staff center-of-mass coordinate movement in the image of continuous 20 frames, if s2<50 millimeter, then from N two field picture, stop gathering the movement locus of staff center of mass point and the information gathering of staff static gesture.
4, the staff center-of-mass coordinate point movement locus collected from M+1 two field picture to N two field picture and staff static gesture are the dynamic action information of the human body gesture collected.
The acquisition of situation contextual information is carried out after unlatching gesture operation function (state of tag=1) always.
The collection of situation contextual information comprises two parts:
1, the feature operation state of intelligent television current time (illustrates, as being in V by the current exchange scenario event performed in this article
0, represent that intelligent television is in normal TV reception state).
2, user figure behavior contextual information: the dynamic gesture trace information and the static gesture information that refer to staff herein.
Step3. mate with human body behavior model basic in human body behavioral data pattern base
While(1)
{
(1) sight event V is in current intelligent television state
0time, under namely intelligent television is in the condition of normal play program state (label tag value is 0), if the figure's behavior PBM now capturing user mates with figure's behavior (WF), then triggered scenario perception events V
1generation; Otherwise intelligent television will remain on and activate V
0state;
(2) context aware event V is at current intelligent television
1during state of activation, under namely intelligent television is in the condition of opening body sense operating function opening (label tag value is 1), if the PBM under current context mates with figure's behavior (WA), then triggered scenario perception events V
2generation, if PBM mates with figure's behavior (WUD), by triggered scenario perception events V
3generation; Otherwise intelligent television will remain on and activate V
1state;
(3) sight event V is in intelligent television state
2time, under namely intelligent television is in the condition of channel mode of operation (label tag value is 2), if the figure's behavior PBM now capturing user mates with figure's behavior (WA), then triggered scenario perception events V
3generation, if the figure's behavior PBM now capturing user mates with figure's behavior (Fist), by triggered scenario perception events V
4generation; Otherwise intelligent television will remain on and activate V
2state;
(4) context aware event V is at current intelligent television
3during generation, under namely intelligent television is in the condition of volume adjusting mode of operation (label value is 3), if the PBM now capturing user mates with figure's behavior (WA), then triggered scenario perception events V
2generation, if the PBM now capturing user mates with figure's behavior (Fist), then triggered scenario perception events V
4generation; Otherwise intelligent television will remain on and activate V
3state;
(5) context aware event V is at current intelligent television
4during generation, under the condition that namely intelligent television is in by closure body sense operating function (label tag value is 4), after completing execution closure body sense operating function, will automatically jump to context aware event V
0state of activation, now label tag value is set to 0, returns step2.
The effect of the inventive method is described below by embodiment:
First, build experiment porch, as shown in Figure 5, comprise figure's behavior extraction and identification module and simulated intelligence tv program module two parts.
Experiment porch configures: in vs2008 development environment, based on MFC class libraries, uses kinect body sense video camera configuration OpenNI platform framework and OpenCV graphics process storehouse.
(1) figure's behavior extraction and identification module
By kinect3D body sense video camera, the depth data of human body can be obtained, the palm of the hand position of hand can be obtained in conjunction with OpenNI platform, and the image of human hands position can be partitioned into.Experimentally requirement, needs the change in depth information gathering the movement locus of dynamic gesture and the modified-image feature of static gesture and gesture.Dynamic gesture path matching algorithm and the static recognizer of gesture is utilized to identify when precursor state behavior model.
Kinect coordinate system belongs to cartesian coordinate system, as shown in Figure 2, wherein, XOY plane is camera plane, if operating personnel are in the face of kinect, X-axis positive direction points to the right side of experimenter, Y-axis positive direction points to experimenter's cephalad direction, Z axis positive direction is consistent with kinect observed direction, Z axis data representation depth information, and the physical unit of Kinect space coordinates is millimeter (mm).In this experiment, according to Kinect, there is perceived depth information capability and obtain the feature of RGB image, extract: the gesture after the RGB image of experimenter, segmentation and the dynamic gesture movement locus characteristic image got and static gesture image, the dynamic gesture movement locus characteristic image got and static gesture image respectively as shown in Figure 3 a and Figure 3 b shows:
(2) simulated intelligence tv program module
The object of design simulation intelligent television program is that the process of the basic operational functions of operative intelligence TV, detects intelligent, the validity of the exchange scenario event perception correlation model proposed in the present invention by simulation actual application environment.Therefore, at this module section, simulated intelligence tv program module mainly simulated intelligence TV normally watches program, and realize channel adjustment, volume adjusting, program broadcast interface dimension scale regulates, the function of gesture operation function and normal play TV programme is closed in open and close.
Channel adjustment function: according to channel list order, realizes the adjustment of a channel program, next channel program.
Volume adjusting function: realize volume and increase or reduce, be adjusted to maximum or till minimum value until volume.
Gesture operation function is closed in open and close: open gesture operation function and represent other feature operation that figure's behavior extraction and identification module and exchange scenario perception correlation model will be utilized to carry out control simulation intelligent television, closes gesture operation function and then represents that figure's behavior extraction and identification module and exchange scenario perception correlation model lose the control to other basic operational functions.
Normal play program function: in this experiment, the state except other basic operational functions all thinks normal play program state, comprises homepage function adjustment state, program just at broadcast state.
Experimental result be analyzed as follows:
The WPU set up herein ?the left-hand component of SmartTV example scenario be figure's behavior extraction and identification module display interface, the human hands dynamic gesture track that what the left side showed from top to bottom successively is extracts and the real world body state RGB image that static gesture image, Kinect device get, the staff RGB image be partitioned into.Right part is simulated intelligence tv program module display interface, intelligent television WPU ?the basic operational functions of SmartTV example all will complete in this part.
For verifying the validity of the inventive method, Stochastic choice 10 experimenters carry out experimental implementation, and gather 1-people's gesture path and gesture motion respectively, everyone often plants basic figure's behavior and do 2 tests, 50 gesture path are wherein as training sample, and all the other 50 as test sample book.First, be familiar with WPU ?SmartTV example laboratory operation rules and basic demand, add up the discrimination of the basic figure's behavior of every experimenter four kinds, and the correct number of times of record basic operation intention perception.Requirement of experiment every experimenter operates four times, and statistics obtains 40 groups of experimental datas altogether.Table 2 gives the discrimination of four kinds of basic figure's behaviors, and table 3 gives user view and infers accuracy.
Table 2
Table 3
The experimental data of table 2, table 3 shows, exchange scenario event correlation sensor model can infer the operation intention of user under user's current state accurately, and the major influence factors of operation intention of user accuracy is the user figure behavior contextual information accurate understanding of current state, namely to the identification of user figure behavior under current state.
By testing above and can finding, when figure's Activity recognition mistake extremely individually or once in a while, based on the body sense intelligent television man-machine interactive system of exchange scenario event correlation sensor model, intelligent television system there will not be the situation of execution error feature operation, and simple carry out operative intelligence television system based on user figure Activity recognition, can because figure's Activity recognition mistake very easily cause the feature operation of intelligent television system execution error.Improve the fault-tolerance of man-machine interaction to a certain extent.
On the basis that the invention discloses application process and principle, be easy to make various types of improvement or distortion, and the method be not limited only to described by the above-mentioned embodiment of the present invention, therefore previously described mode is just preferred, and does not have restrictive meaning.
Claims (9)
1., towards an exchange scenario event correlation Intellisense method for intelligent television application, it is characterized in that:
Catch the historical frames image frames effectively continuously of current K moment exchange scenario state-event contextual information tag and the user figure action trail before the K moment;
Mate with the basic figure's behavior in figure's behavior model pattern database, identify figure's behavior that user is current, pick out the exchange scenario event of current generation accordingly;
Infer according to the exchange scenario event picked out and the exchange scenario event that future time instance will activate generation.
2. the exchange scenario event correlation Intellisense method towards intelligent television application according to claim 1, is characterized in that: described method utilizes kinect3D body sense equipment to obtain the historical frames image frames effectively continuously of the user figure action trail before the K moment.
3. the exchange scenario event correlation Intellisense method towards intelligent television application according to claim 2, is characterized in that: described basic figure's behavior comprises:
WF: wave forward
WA: wave in left and right
WUD: wave up and down
Fist: clench fist.
4. the exchange scenario event correlation Intellisense method towards intelligent television application according to claim 3, is characterized in that: described method comprises:
S1, initialization exchange scenario event correlation relation drives figure;
S2. detect the dynamic action information of human body gesture in the intelligent television broadcast state in K moment and the successive image frame of kinect3D body sense equipment acquisition, catch figure's behavior;
S3, mates with human body behavior model basic in human body behavioral data pattern base, then returns S2.
5. the exchange scenario event correlation Intellisense method towards intelligent television application according to claim 4, is characterized in that: described exchange scenario event correlation relation drives figure as follows:
EDG＝<(WPU-SmartTV，cxt).E，(WPU-SmartTV，cxt).V>；
Wherein, WPU-SmartTV represents that user uses intelligent television to watch the scene of TV programme, and cxt represents situational contexts information;
(WPU-SmartTV，cxt).V＝(V
0，V
1，V
2，V
3，V
4)；
Wherein, V
0program broadcast state sight event, V
1that body sense gesture operation function opens sight event, V
2channel adjustment sight event, V
3volume adjusting sight event, V
4that gesture operation function closes sight event;
(WPU-SmartTV，cxt).E＝(e
01，e
12，e
13，e
22，e
23，e
24，e
32，e
33，e
34，e
41)
Wherein, e
01for WF; e
12, e
32for WUD; e
13, e
23for WA; e
24, e
34for Fist.
6. the exchange scenario event correlation Intellisense method towards intelligent television application according to claim 5, is characterized in that: state S1 and be achieved in that
The basic function status indication parameter of A1, acquisition intelligent television, is input in the array of vertex set (WPU-SmartTV, cxt) .V;
A2, the basic figure's behavior in basic figure's behavior model database is stored in the array of limit collection (WPU-SmartTV, cxt) .E;
A3, arrange the initial effective status value tag=0 of EDG, intelligent television initial condition is V
0.
7. the exchange scenario event correlation Intellisense method towards intelligent television application according to claim 6, is characterized in that: described S2 is achieved in that
B1, obtain the basic function status indication parameter of intelligent television, and assignment is to tag; Tag=0 represents program broadcast state; Tag=1 represents and opens body sense operating function state; Tag=2 represents channel mode of operation; Tag=3 represents volume operation state; Tag=4 represents closure body sense operating function;
B2, as tag=1, obtain the action contextual information of human body gesture: utilize Kinect 3D body sense equipment to obtain human depth's information, obtain the dynamic gesture motion track information of staff, the deep image information that recycling Kinect3D body sense equipment gets and RGB image information, utilize nature staff complexion model to Image Segmentation Using, obtain the static gesture image of nature staff.
8. the exchange scenario event correlation Intellisense method towards intelligent television application according to claim 7, is characterized in that: described B2 comprises:
B21, unlatching Kinect3D body sense equipment, follow the tracks of staff, obtain the center-of-mass coordinate of staff;
In B22, every continuous print M two field picture, calculate the mobile Euclidean distance mean value s1 of staff barycenter in M two field picture; M > 20; If s1 < 50 millimeters, then from the movement locus of center of mass point and the static gesture of staff of M+1 image frame grabber staff;
B23, in the staff gesture contextual information gatherer process from M+1 two field picture, if from N two field picture, the mean value s2 of the Euclidean distance of staff center-of-mass coordinate movement in the image of continuous 20 frames, if s2 < 50 millimeters, then from N two field picture, stop gathering the movement locus of staff center of mass point and the information gathering of staff static gesture; N > M+30;
B24, from M+1 two field picture to N two field picture the staff center-of-mass coordinate point movement locus that collects and staff static gesture be the dynamic action information of the human body gesture collected.
9. the exchange scenario event correlation Intellisense method towards intelligent television application according to claim 8, is characterized in that: carry out mating being achieved in that with human body behavior model basic in human body behavioral data pattern base in described S3
(1) V is in current intelligent television state
0time, if the figure's behavior now capturing user is mated with WF, then trigger V
1generation; Otherwise intelligent television will remain on and activate V
0state;
(2) V is at current intelligent television
1during state of activation, if the figure's behavior under current context is mated with WA, then trigger V
2generation, if figure's behavior is mated with WUD, then trigger V
3generation; Otherwise intelligent television will remain on and activate V
1state;
(3) V is in intelligent television state
2time, if the figure's behavior now capturing user is mated with WA, then trigger V
3generation, if the figure's behavior now capturing user is mated with Fist, then trigger V
4generation; Otherwise intelligent television will remain on and activate V
2state;
(4) V is at current intelligent television
3time, if the figure's behavior now capturing user is mated with WA, then trigger V
2generation, if the figure's behavior now capturing user is mated with Fist, then trigger V
4generation; Otherwise intelligent television will remain on and activate V
3state;
(5) V is at current intelligent television
4time, perform closure body sense operating function if completed, then automatically jump to and activate V
0state, now label tag value is set to 0.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510312111.0A CN105007525A (en) | 2015-06-09 | 2015-06-09 | Interactive situation event correlation smart perception method based on application of smart television |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510312111.0A CN105007525A (en) | 2015-06-09 | 2015-06-09 | Interactive situation event correlation smart perception method based on application of smart television |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN105007525A true CN105007525A (en) | 2015-10-28 |
Family
ID=54380008
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510312111.0A Pending CN105007525A (en) | 2015-06-09 | 2015-06-09 | Interactive situation event correlation smart perception method based on application of smart television |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105007525A (en) |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105915987A (en) * | 2016-04-15 | 2016-08-31 | 济南大学 | Implicit interaction method facing smart television set |
| CN105979358A (en) * | 2016-05-05 | 2016-09-28 | 青岛海信电器股份有限公司 | Volume adjusting method and apparatus and smart terminal |
| CN106682643A (en) * | 2017-01-09 | 2017-05-17 | 济南大学 | Gesture multi-semantic recognition method |
| CN109314722A (en) * | 2016-06-23 | 2019-02-05 | 皇家飞利浦有限公司 | For measuring the method, apparatus and machine readable media of the user's feasibility or ability to accept that are directed to notice |
| CN109582398A (en) * | 2018-11-23 | 2019-04-05 | 阿里巴巴集团控股有限公司 | A kind of condition processing method, device and electronic equipment |
| CN110209394A (en) * | 2019-05-30 | 2019-09-06 | 西安交通大学城市学院 | A kind of individualized intelligent media interface method for building up of cognitive load driving |
| CN110944232A (en) * | 2018-09-21 | 2020-03-31 | 中国移动通信有限公司研究院 | Method and device for monitoring cognitive behaviors and set top box |
| CN114281185A (en) * | 2021-04-25 | 2022-04-05 | 北京壹体体育产业发展有限公司 | Body state recognition and body feeling interaction system and method based on embedded platform |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101515373A (en) * | 2009-03-26 | 2009-08-26 | 浙江大学 | Sports interactive animation producing method |
| CN102854983A (en) * | 2012-09-10 | 2013-01-02 | 中国电子科技集团公司第二十八研究所 | Man-machine interaction method based on gesture recognition |
| US20130263029A1 (en) * | 2012-03-31 | 2013-10-03 | Microsoft Corporation | Instantiable Gesture Objects |
-
2015
- 2015-06-09 CN CN201510312111.0A patent/CN105007525A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101515373A (en) * | 2009-03-26 | 2009-08-26 | 浙江大学 | Sports interactive animation producing method |
| US20130263029A1 (en) * | 2012-03-31 | 2013-10-03 | Microsoft Corporation | Instantiable Gesture Objects |
| CN102854983A (en) * | 2012-09-10 | 2013-01-02 | 中国电子科技集团公司第二十八研究所 | Man-machine interaction method based on gesture recognition |
Non-Patent Citations (3)
| Title |
|---|
| 徐光祐等: "普适计算模式下的人机交互", 《计算机学报》 * |
| 樊银亭等: "基于经验感知的自适应用户界面模型", 《计算机学报》 * |
| 顾君忠: "情景感知计算", 《华东师范大学学报（自然科学版）》 * |
Cited By (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105915987A (en) * | 2016-04-15 | 2016-08-31 | 济南大学 | Implicit interaction method facing smart television set |
| CN105915987B (en) * | 2016-04-15 | 2018-07-06 | 济南大学 | A kind of implicit interactions method towards smart television |
| CN105979358A (en) * | 2016-05-05 | 2016-09-28 | 青岛海信电器股份有限公司 | Volume adjusting method and apparatus and smart terminal |
| CN109314722A (en) * | 2016-06-23 | 2019-02-05 | 皇家飞利浦有限公司 | For measuring the method, apparatus and machine readable media of the user's feasibility or ability to accept that are directed to notice |
| CN106682643A (en) * | 2017-01-09 | 2017-05-17 | 济南大学 | Gesture multi-semantic recognition method |
| CN110944232A (en) * | 2018-09-21 | 2020-03-31 | 中国移动通信有限公司研究院 | Method and device for monitoring cognitive behaviors and set top box |
| CN110944232B (en) * | 2018-09-21 | 2021-11-19 | 中国移动通信有限公司研究院 | Method and device for monitoring cognitive behaviors and set top box |
| CN109582398A (en) * | 2018-11-23 | 2019-04-05 | 阿里巴巴集团控股有限公司 | A kind of condition processing method, device and electronic equipment |
| CN109582398B (en) * | 2018-11-23 | 2022-02-08 | 创新先进技术有限公司 | State processing method and device and electronic equipment |
| CN110209394A (en) * | 2019-05-30 | 2019-09-06 | 西安交通大学城市学院 | A kind of individualized intelligent media interface method for building up of cognitive load driving |
| CN114281185A (en) * | 2021-04-25 | 2022-04-05 | 北京壹体体育产业发展有限公司 | Body state recognition and body feeling interaction system and method based on embedded platform |
| CN114281185B (en) * | 2021-04-25 | 2023-10-27 | 浙江壹体科技有限公司 | Body state identification and somatosensory interaction system and method based on embedded platform |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Jalal et al. | Students’ behavior mining in e-learning environment using cognitive processes with information technologies | |
| CN105007525A (en) | Interactive situation event correlation smart perception method based on application of smart television | |
| WO2021017606A1 (en) | Video processing method and apparatus, and electronic device and storage medium | |
| Materzynska et al. | The jester dataset: A large-scale video dataset of human gestures | |
| WO2021129064A1 (en) | Posture acquisition method and device, and key point coordinate positioning model training method and device | |
| Ehsanpour et al. | Joint learning of social groups, individuals action and sub-group activities in videos | |
| Kale et al. | A study of vision based human motion recognition and analysis | |
| CN105930785B (en) | Intelligent concealed-type interaction system | |
| US20130077820A1 (en) | Machine learning gesture detection | |
| CN111488824A (en) | Motion prompting method and device, electronic equipment and storage medium | |
| CN102426480A (en) | Man-machine interactive system and real-time gesture tracking processing method for same | |
| CN105915987B (en) | A kind of implicit interactions method towards smart television | |
| CN110462684A (en) | Utilize the system of the movement of self-encoding encoder prediction object of interest | |
| US10401947B2 (en) | Method for simulating and controlling virtual sphere in a mobile device | |
| CN110633004A (en) | Interaction method, device and system based on human body posture estimation | |
| Zhang et al. | Handsense: smart multimodal hand gesture recognition based on deep neural networks | |
| CN111104930A (en) | Video processing method and device, electronic equipment and storage medium | |
| de Carvalho et al. | Action recognition for educational proposals applying concepts of Social Assistive Robotics | |
| Zhao et al. | A survey of deep learning in sports applications: Perception, comprehension, and decision | |
| Abhishek et al. | Human Verification over Activity Analysis via Deep Data Mining | |
| CN113220114B (en) | Face recognition-fused embeddable non-contact elevator key interaction method | |
| CN114967937A (en) | Virtual human motion generation method and system | |
| Li et al. | Feature Point Matching for Human-Computer Interaction Multi-Feature Gesture Recognition Based on Virtual Reality VR Technology | |
| CN102591456A (en) | Detection of body and props | |
| Li | Design of human-computer interaction system using gesture recognition algorithm from the perspective of machine learning |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20151028 |