CN107992598A - A kind of method that colony's social networks excavation is carried out based on video data - Google Patents

A kind of method that colony's social networks excavation is carried out based on video data Download PDF

Info

Publication number
CN107992598A
CN107992598A CN201711327006.XA CN201711327006A CN107992598A CN 107992598 A CN107992598 A CN 107992598A CN 201711327006 A CN201711327006 A CN 201711327006A CN 107992598 A CN107992598 A CN 107992598A
Authority
CN
China
Prior art keywords
personage
people
parameters
speech
relation
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201711327006.XA
Other languages
Chinese (zh)
Other versions
CN107992598B (en
Inventor
李大庆
张云轩
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beihang University
Original Assignee
Beihang University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beihang University filed Critical Beihang University
Priority to CN201711327006.XA priority Critical patent/CN107992598B/en
Publication of CN107992598A publication Critical patent/CN107992598A/en
Application granted granted Critical
Publication of CN107992598B publication Critical patent/CN107992598B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/70—Information retrieval; Database structures therefor; File system structures therefor of video data
    • G06F16/78—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
    • G06F16/783—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
    • G06F16/7837—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content using objects detected or recognised in the video content
    • G06F16/784—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content using objects detected or recognised in the video content the detected or recognised objects being people
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q50/00—Systems or methods specially adapted for specific business sectors, e.g. utilities or tourism
    • G06Q50/01—Social networking
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/161—Detection; Localisation; Normalisation

Abstract

The present invention provides a kind of method that colony's social networks excavation is carried out based on video data, its step is as follows：First, character image in video is pre-processed；2nd, the correlation calculations between two people, draw social networks figure, excavate the corresponding group relation of feature subgraph；3rd, the analysis and prediction of group relation；4th, one manual expression and status are analyzed；By above step, present invention mining data from the more rich video data material of content, can solve the practical problem of more people personage's complex relationships in video, so that effectively more relationships in various scenes are quantified and be evaluated；The present invention supports the following real-time analysis to each real life scene, can provide strong method to the character relation analysis in more people's complex scenes and support.

Description

A kind of method that colony's social networks excavation is carried out based on video data
Technical field：
The present invention proposes a kind of method that colony's social networks excavation is carried out based on video data.It is related to face inspection Survey, identification, Crosslinking Structural, calculates social networks etc., belong to Data Mining.
Background technology：
It is a fast-developing field that social relationships, which are excavated, is a cross discipline, has deeply merged computer science With social science.General interpersonal relation is more difficult obtained by the social investigation means such as direct questionnaire, it is necessary to by Present big data means, indirectly carry out social relationships excavation.Social relationships excavate in scientific and technological management, business intelligence, meet an urgent need The many aspects such as management, fight against terrorism and violence have very extensive application.
" social relationships are most important for finding the meaning in living ".Contacted by establishing and keeping with other people, it is a People can obtain it is various happiness (Compton, 2005；Lee＆Robbins, 1998).For example, the identity approval of many members of community It is related with happiness：Including religious organizations (Diener＆Clifton, 2002), group of the same generation of senior middle school (Connolly, White, Stevens and Burnstein, 1987), work/employment (Haughey, 1993) and minority group (Branscombe, Schmitt ,s ＆Harvey, 1999).Therefore, our purpose is exactly based on available data excavating resource and goes out deeper colony Social networks (Daniel L.Wann, Matthew Brasher, Paula J.Waddill, Sagan Ladd).
Present society relation excavation mainly has in the following manner：Traditional questionnaire, online social network analysis, cellular network Network, text data digging, mail network data, travel information and trace information etc..
In terms of data in mobile phone communication, existing document points out (Nathan Eagle, Alex (Sandy) Pentland, and David Lazer,2009)：By mobile phone call history, being based only on observation data can accurately infer that 95% friend is closed System, wherein friends show time and the spatial model of uniqueness in their call model.Review as a result, these behavior moulds Formula is it is also predicted that go out the characteristic of personal aspect, such as satisfaction to work.By collecting the communication information, including mobile phone Position and neighbouring data (including related message registration, the bluetooth equipment near about five meters, phone number, application program make With situation and mobile phone state etc.), and by the behavior social networks of gained compared with the self-report relation from same group, Friendship degree and satisfaction between individual etc. can even be analyzed.
In terms of travel information and track, by taking GeoLife as an example (Yu Zheng, Xing Xie and Wei-Ying Ma, 2010), there is provided three crucial application scenarios：1) adventure in daily life is shared based on GPS track；2) general tourism is suggested, for example, Most interesting place, the travelling order provided in the place of given area with travelling expert；3) personalized friend and position Recommend.In figure is positioned, node is that position (has two kinds of node, i.e. user and position.), it is oriented between two positions Side represents that at least some users continuously travel through the two positions in route.Side right table of the user from starting position to end position Show that user had accessed the number of the position.Further speaking, it is concluded that going out two users have accessed reality The number of same position in the world, and then infer the relation of two people.
In terms of text data：As the popularization that Web 2.0 is applied, more and more Web users are energetically online Issue text message (Qiaozhu Mei, Deng Cai, Duo Zhang, ChengXiang Zhai, 2008).These users Social networks is often formed in a variety of ways, is increased while being reflected in text message and network structure (such as social networks).With Exemplified by blog, the discussion of extensive theme and diversification can be found in blog articles, and between bloger it is fast-developing Friendship network.Researcher periodically publishes thesis, we can not only obtain text message, and can naturally also obtain H3 segment Network.For example, two researchers often cooperated may study same subject, it is therefore more likely that in same research neck Domain.Geographically sensitive event (such as Hurricane Katrina) can also deduce more information, for example, living in adjacent place Blog often writes out similar topic.
Online network facet is often combined with the excavation of above-mentioned text message, for example, being made a speech by forum, commenting on acquisition Deeper time information etc..Including social networking systems such as Email, Twitter, blog, microblogging, forums.For example, there is text Chapter inquires into (Christopher P.Diehl, Jaime Montemayor, Mike Pekala, 2009)：How is the fan that climbs the mountain The contact between bloger and reader can be found using technology is currently available thatIt is it is desirable that of interest from mountain-climbing fan In bloger, some styles and the individual character bloger similar to the fan that climbs the mountain are found.Our only use information clue and knots at present Structure clue determines analogical object.Therefore, the burden of other mountain-climbings fan bloger is found, still with active searchers, Active searchers also needs to further remove social, participation correlated activation.Therefore, a Social search engine is contemplated in article, These digital society products are analyzed, and show the timeline of relation between each blog of description.These timelines present The dependency relation especially active period.Examine these as a result, and adjust the period of suggestion, cover sought by active searchers Relation period.Social search engine is found in the language of blog and interaction style and distinguishes different social letters Number.These characteristic signals are used to sequence and climb bloger, and determine its specific position, show similar to active searchers institute Desired interaction style.
The example of another online social networks be Enron scandal event investigation (Christopher P.Diehl, Jaime Montemayor, Mike Pekala, 2009).Electronics finds that industry has occurred, and is in and provides technical support During.When major company is subject to investigation to examine, substantial amounts of file, Email and other digital contents can be retrieved analysis, For consummating legal case.In the fact-finding process of U.S. government, involved file also becomes a part for public records. Covered in investigation about 250 in about 150 safe electronic mail accounts, 000 different Email, and draw out spy Sign figure is judged.For example, being drawn the address of mail as node in plan view, all communications are subjected to line, are painted Self-centered social relationships identification figure etc. is made, has all referred to the data mining of online social networks.
Complex network can also be applied to reply terrorism, many people including terrorism with crowd's social networks Class collective activity has been demonstrated there is universal pattern (Juan Camilo Bohorquez, Sean Gourley, Alexander R.Dixon, Michael Spagat＆Neil F.Johnson, 2009) i.e.：In each war from 1816 to 1980, wound The scale distribution for dying number and the attack of terrorism is proved to follow the rule similar to power, the relevant distribution of law.And There are unified mankind's armed rebellion pattern, these common ground, and the change of the angle specific explanations conflict from quantization can be reproduced Change.It is another to there is research (Li Benxian, Li Mengjun, side's brocade is clear, faces upward Jin and admires, 2013) to find:During network evolution, terroristic organization Network evolution not only have worldlet, the hierarchical organization of uncalibrated visual servo characteristic concurrently, also using the Centroid of different stage by network It is connected as a single entity, and there is an important factor for promoting terrorist network constantly to develop, the i.e. increase with terrorist's conviction and environment The growth of resource is related；Therefore, the selectivity strike of anti-terrorism side will increase inhibited and certain control to network Effect.
In conclusion current research and algorithm are based on data types such as quantized character or language analyses, into the ranks mostly The social networks connect are excavated and analysis, but the social connection aspect more top layer that these data are embodied, and are in social relationships mostly Long-term static relation accumulation (such as address book data), or the embodiment (such as track data) of life pattern similitude, or certain Social activity (such as microblogging or blog) under one interest, it is more difficult to be embodied in a colony by polymorphic type, more interest and more behavior patterns The further feature of middle social networks.For example there is different interest, the people of habits and customs, also can be usually good friend, and Show abundant group activity and relationship type.How to excavate more relationships in these activities be dynamically composed and pilot bus It is that we excavate a key issue in social relationships deep content.Particularly this main social activity is talked or discussed in colony How activity, therefrom excavate the main contents that social relationships are the present invention.
Present invention is generally directed to include the video data of crowd's conversation on course, carry out the human relation based on recognition of face and dig Pick, analysis result can be found that the potential social sub-network in crowd, and right of speech grasp person therein.Specific main basis The various dynamic parameters of character face, form the correlation metric between two people, build correlation networks on this basis, go forward side by side One step analyzes network characterization, and each socialgram of comprehensive analysis and each feature subgraph structure carry out the pre- of relation between more people Survey, draw the deep relationship network of personage appeared in video.For example, active interactive Duo Ren groups find that right of speech is grasped Relation of person and other personages etc..Used vedio data compared to numeral and the data of the other forms such as text, Intension is more abundant, instant, the social relationships excavated more multidimensional, embodies the deep structure of crowd's interaction.
The content of the invention
(1) purpose of invention
The purpose of the present invention is：The present invention provides a kind of side that colony's social networks excavation is carried out based on video data Method, effectively can be quantified and be evaluated to more human world character relations in video.
The theoretical foundation of the present invention：The behavior that interpersonal social relationships intensity is shown as to a certain extent is related Property.By social influence, the behavior of people can guide his friend to act (Aris in a similar way Anagnostopoulos,Ravi Kumar,Mohammad Mahdian,2008).Therefore can be according to face in conversation on course Dynamic correlation between face, judges social networks intensity and type between two people, and then forms the society between more people Hand over Web Mining.On this basis, according to the appearance situation of socialgram in each network come more active small in discovery crowd Colony, and then deduce out the complicated matching relationship between more people.Line segment between personage has directive property, it can be determined that crowd begs for By or leader in talking, distinguish between personage actively and passive relation.
(2) technical solution
The technical solution of the present invention：Establish structure and the analysis of colony's social networks figure based on video data.It is first First, Face datection pretreatment is carried out, pretreatment zone is divided into two parts, i.e. hair and face up and down.Related pixel is extracted, is carried out Dyeing, counts, analysis.Secondly, carry out the calculating of correlation to the relation between every two people, the result of correlation be used for setting-out, Draw socialgram.Line between two people includes the statistical information such as conspicuousness, directive property.3rd, excavation, the analysis of group relation And prediction, including the excavation of each feature subgraph and the excavation of key person.For example, group relation may close for leader-subordinate System, group-antagonistic relations, more friends, close relationships etc..Finally, for single personage, according to group relation socialgram Difference, assignment carried out to given parameters in series and is sorted, and then may determine that each personage characteristic of oneself, comprising dynamic The feature of work, language, degree of participation and leading degree etc. several aspects.
A kind of method that colony's social networks excavation is carried out based on video data of the present invention, its step are as follows：
Step 1: character image in video is pre-processed；
The present invention is based on meeting each video material data claimed below：
(a) facial information：It is satisfied with Face datection condition, it is impossible to be that the figure viewed from behind or whole process are in the angle of rotation that can't detect Degree, face off and on can use；
(b) temporal information：Length, clarity and the frame number of video；Wherein clarity meets that basic Face datection is differentiated Rate；
First, Face datection is carried out to each frame in video, dilatation, increase is carried out to obtained square face mask Hair portion, and small rectangle is drawn in two parts respectively, its size and location is determined by the profile size of Face datection；
Secondly, each pixel in small rectangle is averaged, the average value of three kinds of colors is respectively obtained, as pixel The reference value c of scale modelb、cg、cr；To realize the effect for automatically giving reference value；The reason is that the picture of head and face Element possesses specific color data ratio in many scenes, and three kinds of colors have certain metastable proportionate relationship, can Suitable for different scenes, and under different light keep proportionate relationship stabilization；But proportionate relationship is based on reference value cb、cg、 cr, and reference value cb、cg、crOtherness is very strong in varying environment difference light；Therefore, reference value is usual in testing in the past To measure in advance, then carry out the test under identical environment, to realize the dyeing of hair and face；Dyeing refers to that handle meets ratio The point of example model dyes red and purple (replaceable), represents hair and face respectively；Therefore, the automatic inspection of small rectangle is set , to ensure in each frame, even if light is different, there are scene changes, object pixel can also be contaminated automatically in survey mechanism Color；
3rd, we have obtained hair and have dyed red, the face of facial purple；We unite pixel quantity Meter, contrasts the situation of change of each frame, if that is, red first increases and then decreases, and change and exceed certain threshold m, then it has been determined as Into an elemental motion；If purple first increases and then decreases, and change and exceed certain threshold m, then it is judged to completing once basic Action；
4th, we give changing value δ h to complete once the Hs parameters of the personage of above-mentioned elemental motion；Increase at the same time another The variable of two affecting parameters Hs, one be Face datection frame (dete frames) offset, for detecting significantly face Displacement, if the situation of change of continuous n frame coordinate offsets meets given rule, and variable quantity reaches the threshold s of setting, Then judge that personage completes an elemental motion；The variable of another affecting parameters Hs is the lasting frame number of the dete frames to disappear, For solving the problems, such as that Face datection frame off and on brings interference；Our solution is：Frame within m frames disappears Its original continuity is kept, if still continued depletion after m frames, its intensity reduce, gives Hs parameter small change value δ h, Show decline trend；
Finally, we are by the total variation A of the activity coefficient Hs of each personage of each framer(d) (solved in step 2 Release) deposit two-dimensional array, is called when to be calculated；
Step 2: the correlation calculations between two people, draw social networks figure, excavate the corresponding colony of feature subgraph and close System；
1. one active value；
In video using every k frames as interval of time, d represents the d frames in k frame periods；
Ar(d)=Σn(δh)
Its implication is in d frames, and personage completes the changing value added up after the elemental motion of quantity n time, and δ h are represented every time The corresponding different changing values of different elemental motions are completed, r numbers for personage, and d represents d frames；
Ar y arv={ ΣD=1 k(Ar(d))}/k
Its implication is A in y sections of time intervalsr(d) average value of parameter；Since d=1, terminate to d=k, y is represented Y sections of time intervals；R numbers for personage, and d represents d frames；
2. the correlation calculations between every two people；
A correlation calculations (Cross Correlation) is done between each two personage；
Fτ y (l,r)={ ΣD=1 (k-t)|(Al y(d)-Al y arv)(Ar y(d+τ)-Ar y arv)|}/(k-τ)
(τ>0) when；
Fτ y (l,r)={ Σd=1(k+t)|(Al y(d-τ)-Al y arv)(Ar y(d)-Ar y arv)|}/(k+τ)
(τ<0) when；
Fτ y (l,r)(τ<0)≡Fτ y (r,l)(τ>0)
Fτ y (l,r)Implication in y sections of time intervals, correlation values summation of (l, r) two people when the time difference is τ Average value；The positive and negative of wherein τ is used to judge directive property, τ>0 and τ<0 arrow for representing two different directions respectively is directed toward；
Fmax y (l,r)=max (Fτ y (l,r)),(-k<τ<k)
Fmax y (l,r)Represent in the range of the y periods, time difference τ ∈ (- k, k), the corresponding difference of different τ values Fτ y (l,r)In, the maximum in two people's correlation values sum-total averages is filtered out, and retain its corresponding τ value and be used to judge arrow Head is directed toward (such as Fig. 2, Fig. 3)；
3. draw social networks figure (referred to as " socialgram ")；
To the maximal correlation property coefficient F in two human world all in groupmax y (l,r)It is ranked up, sets selection condition, according with Setting-out is carried out between all (l, r) two people of conjunction condition, is depicted as the socialgram of y sections of time intervals；Socialgram is that group closes It is the basis excavated；
4. excavate the corresponding group of feature subgraph；
Socialgram can be split as the feature subgraph such as ray vertex, triangle, star-like, quadrangle, pentagon；Main feature Subgraph is described below：
Ray vertex structure, such as Fig. 4, i.e. two rays have same vertices；Corresponding Tn parameters；
There is line between triangular structure, such as Fig. 5, Fig. 6, i.e. three people, form triangle；Corresponding Tr parameters；
The expansion of star structure, such as Fig. 7, i.e. ray vertex structure, refers to a plurality of ray and meets at a bit；Corresponding Tn parameters；
Linear structure, such as Fig. 8, represent the line between two people, have generality, appear in each scene；Straight line knot Structure includes straight line connection of taking advantage of a situation, and such as Fig. 9, refers to the line between each personage and finally only form a broken line；Corresponding Lt parameters；
The corresponding group of each feature subgraph of comprehensive analysis, preparation is provided for the analysis and prediction of group relation.
Step 3: the analysis and prediction of group relation；
1. define correlated variables：
We define Te, Tr, Tn, Lt, Ct, Hs, R, Ji, Jt, 9 variables and are used for ensuing calculating；Wherein Ct is Matrix, other specification are one-dimension arrays, each personage's numbering l, r etc. in matrix video corresponding with the sequence number in array；
1) Te parameters reflect speech efficiency, are initially 0, if occur every time it is star-like after limited frame in occur not including The triangle of this personage, increases the variable quantity δ n of personage's Te parameters at this time；Another situation is, when statistics occurs star-like every time this The outer directional arrow quantity of personage, and increase the variable quantity δ n of corresponding Te values；
2) Tr parameters reflect participation, are initially 0, when occurring triangle every time, the Tr of the appearance personnel in triangle Value change δ n；
3) Tn parameters correspond to speech number, are initially 0, when occurring star-like every time, the Tn values change δ n of apex personage；
4) it is not completely independent between the frequent degree and amplitude of Hs parameters respective action, with speech number Tn；Hs=ΣI=1 f Ar(d)；Wherein f is video totalframes；Assignment is carried out according to each elemental motion situation is completed in each frame of pretreatment, if reaching To standard, then Hs parameters increase Ar(d) variable quantity；
5) parameter R=Te/Tn；Meaning directly perceived is the effective ratio of speech；Its effect is similar but not exactly the same with Te；
6) Ji, Jt are arrow directive property parameter, and according to the directive property of every line segment, Ji, Jt value of each personage is become Change δ n；If arrow is outer direction, such as Figure 10, then Jt Parameters variations δ n, if arrow is interior direction, such as Figure 11, then Ji Parameters variations δ n；It should be noted that Jt also determines Te parameters, and when personage meets to appear in the condition on star-like vertex, Te=Te+Jt (d)； Wherein Jt (d) represents the Jt values of d frames；D represents frame number；
7) Ct and Lt parameters reflect in group reciprocation degree between two people；Lt parameters represent the line of two people Number, if there is line, numerical value change δ n；Ct matrixes represent the speech level of interaction of two people, its computational methods is the company of extracting Continuous star-like sequence, front and rear two personages continuously occurred in sequence are considered as and are once interacted, are stored in the matrix position of reference numeral Put, such as (l, r), matrix numerical value is ranked up, filters out relation between the strong or weak personage of level of interaction in colony；
To sum up, summary of parameters.An can be three series by we, be respectively used to judge group interaction degree, dominance, action Frequency and amplitude；
2. the analysis method that group relation is excavated；
First, Ji is analyzed, Jt frequency distribution, excavates the stronger personage of prime move in group relation with being inclined to passive people Thing；Secondly, it is ranked up by Tn, Hs frequency distribution, number of talking to each personage with there is operating frequency；3rd, Te is analyzed, Tr, R frequency distribution, the speech effect to each personage are ranked up with degree of participation；4th, by Lt frequency distribution, to two The power of relationship is ranked up；Finally, Ct matrix datas are analyzed, talk between personage degree of cooperation, interactive degree are arranged Sequence；
3. group relation excavates prediction；
We integrate all the above information and four people's scene group relations in example (order from left to right) are predicted：
In terms of group interaction：There is a small amount of talk between No. 1 and No. 2 personages, there is simple talk, and work well, interactive degree Height, degree of cooperation are high；Compared to No. 1 personage of No. 2 personages is closer with No. 3 No. 4 character relations；No. 2 No. 4 personages are more likely to locate In talk situation, wherein No. 2 more active, 4 numbers it；No. 3 personage's participations are very high, and speech action effect is higher, but talk and live Jerk is small, may infer that for speech content it is attractive, it is not boring；In addition, the reciprocal in No. 3 No. 4 two human world is also very good, Illustrate that two people have communication process, there is potential tacit agreement in relation or topic；
In terms of dominance：No. 2 No. 4 obvious higher, No. 4 personage's highests；No. 4 personages were both talkative, and speech effect is again high, participates in Degree is again good, therefore predicts that dominance is stronger compared with No. 2；For No. 2 personages there are certain contradiction, i.e., existing dominance, there is necessarily quilt Ejector half；
In terms of action：No. 1 No. 2 actions are on the high side, it is presumed that No. 3 No. 4 rare headworks, though also No. 1 participation of deducibility Spend low, but be not at departing from colony state；
Four people's scenes that our trials are surveyed with language reduction：1, No. 2 personage, 2, No. 3 personages, between 3, No. 4 personages point There is not good communication process, there are closer talk；Overall four relationships are close, and a kind of is probably to be heightened in spirits between friend Chat scenario, for intimate degree, can determine that most people is friends, higher than strange relationship；Another is probably No. 1 No. 2 are antagonistic relations with other people, but are analyzed from status and attitude angle, its prestige is very high, is likely to be at negotiation scene, or open Can scene；
In conclusion largely tallying with the actual situation in prediction result, in real scene, No. 1 is mother, and No. 2 are daughters, No. 3 are sons, and No. 4 are father (order from left to right)；Four people are in visits old man in hospital, in corridor chat scenario, wherein Talk atmosphere is humorous, and the cooperation between personage also complies with infers hypothesis above；And provide final conspicuousness setting-out (such as Figure 12, figure 13)。
Step 4: one manual expression is analyzed with status：
From group relation is analyzed and predicted, we can further extract one manual expression and character trait；
1. single stunt expresses；
First, it is determined that whether personage is active, whether headwork (such as nod or related limb action) is more；Secondly, divide Analyse the number (numerical value does not definitely represent specific number), whether talkative of personage's speech；3rd, judge personage speech whether effect Whether height, action effect are preferable, if other people are had an impact, speech plays a role；Finally, judge that the prime move of personage is high Low, behavior is partially actively or passive；
In example, No. 1 personage has a small amount of speech generally in the state of listening attentively to；No. 2 personage's speeches are more, in language and move Performance is more active in terms of work, and the possible tone is humorous, and possible content is pertinent, or speech of summarizing, and can induce one in a word Enter victory；No. 3 personages also have a small amount of time to participate in into talk, and give opinion, but relatively quiet, and liveness is lower slightly；No. 4 personages say It is less slightly to talk about the time；
2. single status prediction：
No. 4 are more likely much-admired personages or elder, are carried weight, strong, may followed by No. 1 personage Strength is relatively strong but less than speech table；No. 2 No. 3 personages are probably preceding two people subordinate, or prestige is lower slightly, the lower personage of posture；Wherein 2 Number more it is good at active atmosphere, speech is on the high side, but is not belonging to useless speech, may deliver viewpoint or adjust atmosphere；Other No. 2 people Thing embodies the personality of more contradiction compared with No. 3 character personalities, is more likely to that individual character is stronger, or there is the inside and outside mood of polarization；
To sum up, largely tally with the actual situation in prediction result；In real scene, No. 1 is mother, and No. 2 are daughters, No. 3 It is son, No. 4 are father；Four people are in visits old man in hospital, in corridor chat scenario, wherein talk atmosphere humour, personage Between cooperation also comply with above infer assume；And provide final conspicuousness setting-out (such as Figure 12, Figure 13).
Wherein, the methods of Face datection, calculating pixel average, graphing, only retain and calculate required information, its Method category known technology, the present invention do not repeat；
By above step, present invention mining data from the more rich video data material of content can be to more extensive Video material is analyzed and processed, and is such as applied in friend's chat, discussion, commercial negotiation different scenes；Support real-time Calculate, the data that computer can be identified and handled can be converted into；Comprehensive analysis is being carried out more on various dimensions, is recording group relation Change, solves the problems, such as relation variability, obtains granulating smaller, more deep crowd's interactive relation；So as to solve video In more people personage's complex relationships practical problem, effectively more relationships in various scenes are quantified and are evaluated；This hair It is bright to support the following real-time analysis to each real life scene, the character relation analysis in more people's complex scenes can be provided strong Powerful method supports.
(3) advantage and effect
Conventional method is compared to, analysis method of the present invention has following advantage：
(a) generality：Breach and be only limitted to numeral or written historical materials etc. in the past, for the more rich video data element of content Material is analyzed.The change such as the scene in video material, light is not limited in use, such as camera lens switching, replaces scene After need not reset parameter, can be automatically extracted.
(b) feature：The social networks excavated can reflect social networks (such as friend in different groups Activity Type Chat, discussion, commercial negotiation etc.), smaller is granulated, more profoundly embodies the interaction pass of people in different environments System.
(c) real-time：Algorithm is simple, can support to calculate in real time.By the complex relationship between obscure abstract personage, need Thinking by people goes to understand the relation analyzed, and is converted into the identifiable signal of computer, computer can be by each feature Subgraph is identified to judge the network of personal connections of personage's recessiveness in video.
(d) multi-dimensional nature：Group relation is more comprehensively, more rich as obtained by image, to group relation in more dimensions Comprehensive analysis is carried out, and records the relationship change between group, solves the problems, such as relation variability.
To sum up, when the result of study of this new method will be to Computer Image Processing, the analysis of machine vision, which provides, to be had by force The method support of power.
Brief description of the drawings：
Fig. 1 the method for the invention flow diagrams；
The arrow directive property schematic diagram of Fig. 2 " socialgram " of the present invention；
The arrow directive property schematic diagram of Fig. 3 " socialgram " of the present invention；
The ray vertex structure of Fig. 4 " socialgram " of the present invention；
The triangular structure of Fig. 5 " socialgram " of the present invention；
The triangular structure of Fig. 6 " socialgram " of the present invention；
The star structure of Fig. 7 " socialgram " of the present invention；
The linear structure of Fig. 8 " socialgram " of the present invention；
The linear structure of taking advantage of a situation of Fig. 9 " socialgram " of the present invention；
The arrow of Figure 10 " socialgram " of the present invention is directed to outside；
The arrow of Figure 11 " socialgram " of the present invention is directed to inside；
The final conspicuousness design sketch of Figure 12 " socialgram " of the present invention；
The final conspicuousness design sketch of Figure 13 " socialgram " of the present invention；
Embodiment
To make the technical problem to be solved in the present invention, technical solution clearer, below in conjunction with attached drawing and specific implementation Case is described in detail.
A kind of method that crowd's social networks excavation is carried out based on video data of the present invention, as shown in Figure 1, its specific steps It is as follows：
Step 1: character image in video is pre-processed；
The present invention is based on meeting each video material data claimed below：
(a) facial information：It is satisfied with Face datection condition, it is impossible to be that the figure viewed from behind or whole process are in the angle of rotation that can't detect Degree, face off and on can use；
(b) temporal information：The length of video, clarity, frame number；Wherein clarity meets basic Face datection resolution ratio ；
First, Face datection is carried out to each frame in video, dilatation, increase is carried out to obtained square face mask Hair portion, and small rectangle is drawn in two parts respectively, its size and location is determined by the profile size of Face datection；
Secondly, each pixel in small rectangle is averaged, the average value of three kinds of colors is respectively obtained, as pixel The reference value c of scale modelb、cg、cr；To realize the effect for automatically giving reference value；The reason is that the picture of head and face Element possesses specific color data ratio in many scenes, and three kinds of first colors have certain metastable proportionate relationship, can Suitable for different scenes, and under different light keep proportionate relationship stabilization；But proportionate relationship is based on reference value cb、cg、 cr, and reference value cb、cg、crOtherness is very strong in varying environment difference light；Therefore, reference value is usual in testing in the past To measure in advance, then carry out the test under identical environment, to realize the dyeing of hair and face；Dyeing refers to that handle meets ratio The point of example model dyes red and purple (replaceable), represents hair and face respectively；Therefore, the automatic inspection of small rectangle is set , to ensure in each frame, even if light is different, there are scene changes, object pixel can also be contaminated automatically in survey mechanism Color；
3rd, we have obtained hair and have dyed red, the face of facial purple；We unite pixel quantity Meter, contrasts the situation of change of each frame, if that is, red first increases and then decreases, and change and exceed certain threshold m, then it has been determined as Into an elemental motion；If purple first increases and then decreases, and change and exceed certain threshold m, then it is judged to completing once basic Action；
4th, we give changing value δ h to complete once the Hs parameters of the personage of above-mentioned elemental motion；Increase at the same time another The variable of two affecting parameters Hs, one be Face datection frame (dete frames) offset, for detecting significantly face Displacement, if the situation of change of continuous n frame coordinate offsets meets given rule, and variable quantity reaches the threshold s of setting, Then judge that personage completes an elemental motion；The variable of another affecting parameters Hs is the lasting frame number of the dete frames to disappear, For solving the problems, such as that Face datection frame off and on brings interference；Our solution is：Frame within m frames disappears Its original continuity is kept, if still continued depletion after m frames, its intensity reduce, gives Hs parameter small change value δ h, Show decline trend；
Finally, we are by the total variation A of the activity coefficient Hs of each personage of each framer(d) (solved in step 2 Release) deposit two-dimensional array, is called when to be calculated；
Step 2: the correlation calculations between two people, draw social networks figure, excavate the corresponding colony of feature subgraph and close System；
1. one active value；
In video using every k frames as interval of time, d represents the d frames in k frame periods；
Ar(d)=Σn(δh)
Its implication is in d frames, and personage completes the changing value added up after the elemental motion of quantity n time, and δ h are represented every time The corresponding different changing values of different elemental motions are completed, r numbers for personage, and d represents d frames；
Ar y arv={ ΣD=1 k(Ar(d))}/k
Its implication is A in y sections of time intervalsr(d) average value of parameter；Since d=1, terminate to d=k, y is represented Y sections of time intervals；R numbers for personage, and d represents d frames；
2. the correlation calculations between every two people；
A correlation calculations (Cross Correlation) is done between each two personage；
Fτ y (l,r)={ ΣD=1 (k-t)|(Al y(d)-Al y arv)(Ar y(d+τ)-Ar y arv)|}/(k-τ)
(τ>0) when；
Fτ y (l,r)={ ΣD=1 (k+t)|(Al y(d-τ)-Al y arv)(Ar y(d)-Ar y arv)|}/(k+τ)
(τ<0) when；
Fτ y (l,r)(τ<0)≡Fτ y (r,l)(τ>0)
Fτ y (l,r)Implication in y sections of time intervals, correlation values summation of (l, r) two people when the time difference is τ Average value；The positive and negative of wherein τ is used to judge directive property, τ>0 and τ<0 arrow for representing two different directions respectively is directed toward；
Fmax y (l,r)=max (Fτ y(l,r)),(-k<τ<k)
Fmax y (l,r)Represent in the range of the y periods, time difference τ ∈ (- k, k), the corresponding difference of different τ values Fτ y (l,r)In, the maximum in two people's correlation values sum-total averages is filtered out, and retain its corresponding τ value and be used to judge arrow Head is directed toward (such as Fig. 2, Fig. 3)；
3. draw social networks figure (referred to as " socialgram ")；
To the maximal correlation property coefficient F in two human world all in groupmax y (l,r)It is ranked up, sets selection condition, according with Setting-out is carried out between all (l, r) two people of conjunction condition, is depicted as the socialgram of y sections of time intervals；Socialgram is that group closes It is the basis excavated；
4. excavate the corresponding group of feature subgraph；
Socialgram can be split as the feature subgraph such as ray vertex, triangle, star-like, quadrangle, pentagon；Main feature Subgraph is described below：
Ray vertex structure, such as Fig. 4, i.e. two rays have same vertices；Corresponding Tn parameters；
There is line between triangular structure, such as Fig. 5, Fig. 6, i.e. three people, form triangle；Corresponding Tr parameters；
The expansion of star structure, such as Fig. 7, i.e. ray vertex structure, refers to a plurality of ray and meets at a bit；Corresponding Tn parameters；
Linear structure, such as Fig. 8, represent the line between two people, have generality, appear in each scene；Straight line knot Structure includes straight line connection of taking advantage of a situation, and such as Fig. 9, refers to the line between each personage and finally only form a broken line；Corresponding Lt parameters；
The corresponding group of each feature subgraph of comprehensive analysis, preparation is provided for the analysis and prediction of group relation.
Step 3: the analysis and prediction of group relation；
1. define correlated variables：
We define Te, Tr, Tn, Lt, Ct, Hs, R, Ji, Jt, 9 variables and are used for ensuing calculating；Wherein Ct is Matrix, other specification are one-dimension arrays, each personage's numbering l, r etc. in matrix video corresponding with the sequence number in array；
7) Te parameters reflect speech efficiency, are initially 0, if occur every time it is star-like after limited frame in occur not including The triangle of this personage, increases the variable quantity δ n of personage's Te parameters at this time；Another situation is, when statistics occurs star-like every time this The outer directional arrow quantity of personage, and increase the variable quantity δ n of corresponding Te values；
8) Tr parameters reflect participation, are initially 0, when occurring triangle every time, the Tr of the appearance personnel in triangle Value change δ n；
9) Tn parameters correspond to speech number, are initially 0, when occurring star-like every time, the Tn values change δ n of apex personage；
10) it is not completely independent between the frequent degree and amplitude of Hs parameters respective action, with speech number Tn；Hs= ΣI=1 f Ar(d)；Wherein f is video totalframes；Assigned according to each elemental motion situation is completed in each frame of pretreatment Value, if reaching standard, Hs parameters increase Ar(d) variable quantity；
11) parameter R=Te/Tn；Meaning directly perceived is the effective ratio of speech；Its effect is similar but not exactly the same with Te；
12) Ji, Jt are arrow directive property parameter, and according to the directive property of every line segment, Ji, Jt value of each personage is become Change δ n；If arrow is outer direction, such as Figure 10, then Jt Parameters variations δ n, if arrow is interior direction, such as Figure 11, then Ji Parameters variations δ n；It should be noted that Jt also determines Te parameters, and when personage meets to appear in the condition on star-like vertex, Te=Te+Jt(d)；Its Middle Jt(d)Represent the Jt values of d frames；D represents frame number；
7) Ct and Lt parameters reflect in group reciprocation degree between two people；Lt parameters represent the line of two people Number, if there is line, numerical value change δ n；Ct matrixes represent the speech level of interaction of two people, its computational methods is the company of extracting Continuous star-like sequence, front and rear two personages continuously occurred in sequence are considered as and are once interacted, are stored in the matrix position of reference numeral Put, such as (l, r), matrix numerical value is ranked up, filters out relation between the strong or weak personage of level of interaction in colony；
To sum up, summary of parameters.An can be three series by we, be respectively used to judge group interaction degree, dominance, action Frequency and amplitude；
2. the analysis method that group relation is excavated；
First, Ji is analyzed, Jt frequency distribution, excavates the stronger personage of prime move in group relation with being inclined to passive people Thing；Secondly, it is ranked up by Tn, Hs frequency distribution, number of talking to each personage with there is operating frequency；3rd, Te is analyzed, Tr, R frequency distribution, the speech effect to each personage are ranked up with degree of participation；4th, by Lt frequency distribution, to two The power of relationship is ranked up；Finally, Ct matrix datas are analyzed, talk between personage degree of cooperation, interactive degree are arranged Sequence；
3. group relation excavates prediction；
We integrate all the above information and four people's scene group relations in example (order from left to right) are predicted：
In terms of group interaction：There is a small amount of talk between No. 1 and No. 2 personages, there is simple talk, and work well, interactive degree Height, degree of cooperation are high；Compared to No. 1 personage of No. 2 personages is closer with No. 3 No. 4 character relations；No. 2 No. 4 personages are more likely to locate In talk situation, wherein No. 2 more active, 4 numbers it；No. 3 personage's participations are very high, and speech action effect is higher, but talk and live Jerk is small, may infer that for speech content it is attractive, it is not boring；In addition, the reciprocal in No. 3 No. 4 two human world is also very good, Illustrate that two people have communication process, there is potential tacit agreement in relation or topic；
In terms of dominance：No. 2 No. 4 obvious higher, No. 4 personage's highests；No. 4 personages were both talkative, and speech effect is again high, participates in Degree is again good, therefore predicts that dominance is stronger compared with No. 2；For No. 2 personages there are certain contradiction, i.e., existing dominance, there is necessarily quilt Ejector half；
In terms of action：No. 1 No. 2 actions are on the high side, it is presumed that No. 3 No. 4 rare headworks, though also No. 1 participation of deducibility Spend low, but be not at departing from colony state；
Four people's scenes that our trials are surveyed with language reduction：1, No. 2 personage, 2, No. 3 personages, between 3, No. 4 personages point There is not good communication process, there are closer talk；Overall four relationships are close, and a kind of is probably to be heightened in spirits between friend Chat scenario, for intimate degree, can determine that most people is friends, higher than strange relationship；Another is probably No. 1 No. 2 are antagonistic relations with other people, but are analyzed from status and attitude angle, its prestige is very high, is likely to be at negotiation scene, or open Can scene；
In conclusion largely tallying with the actual situation in prediction result, in real scene, No. 1 is mother, and No. 2 are daughters, No. 3 are sons, and No. 4 are father (order from left to right)；Four people are in visits old man in hospital, in corridor chat scenario, wherein Talk atmosphere is humorous, and the cooperation between personage also complies with infers hypothesis above；And provide final conspicuousness setting-out (such as Figure 12, figure 13)。
Step 4: one manual expression is analyzed with status：
From group relation is analyzed and predicted, we can further extract one manual expression and character trait；
1. single stunt expresses；
First, it is determined that whether personage is active, whether headwork (such as nod or related limb action) is more；Secondly, divide Analyse the number (numerical value does not definitely represent specific number), whether talkative of personage's speech；3rd, judge personage speech whether effect Whether height, action effect are preferable, if other people are had an impact, speech plays a role；Finally, judge that the prime move of personage is high Low, behavior is partially actively or passive；
In example, No. 1 personage has a small amount of speech generally in the state of listening attentively to；No. 2 personage's speeches are more, in language and move Performance is more active in terms of work, and the possible tone is humorous, and possible content is pertinent, or speech of summarizing, and can induce one in a word Enter victory；No. 3 personages also have a small amount of time to participate in into talk, and give opinion, but relatively quiet, and liveness is lower slightly；No. 4 personages say It is less slightly to talk about the time；
2. single status prediction：
No. 4 are more likely much-admired personages or elder, are carried weight, strong, may followed by No. 1 personage Strength is relatively strong but less than speech table；No. 2 No. 3 personages are probably preceding two people subordinate, or prestige is lower slightly, the lower personage of posture；Wherein 2 Number more it is good at active atmosphere, speech is on the high side, but is not belonging to useless speech, may deliver viewpoint or adjust atmosphere；Other No. 2 people Thing embodies the personality of more contradiction compared with No. 3 character personalities, is more likely to that individual character is stronger, or there is the inside and outside mood of polarization；
To sum up, largely tally with the actual situation in prediction result；In real scene, No. 1 is mother, and No. 2 are daughters, No. 3 It is son, No. 4 are father；Four people are in visits old man in hospital, in corridor chat scenario, wherein talk atmosphere humour, personage Between cooperation also comply with above infer assume；And provide final conspicuousness setting-out (such as Figure 12, Figure 13).
Non-elaborated part of the present invention belongs to techniques well known.
The above, is only part embodiment of the present invention, but protection scope of the present invention is not limited thereto, and is appointed What those skilled in the art the invention discloses technical scope in, the change or replacement that can readily occur in should all be covered Within protection scope of the present invention.

Claims (1)

  1. A kind of 1. method that colony's social networks excavation is carried out based on video data, it is characterised in that：Its step is as follows：
    Step 1: character image in video is pre-processed；
    The present invention is based on meeting each video material data claimed below：
    (a) facial information：It is satisfied with Face datection condition, it is impossible to it is that the figure viewed from behind and whole process are in the rotational angle that can't detect, when The face continued when disconnected can use；
    (b) temporal information：Length, clarity and the frame number of video；Wherein clarity meets basic Face datection resolution ratio i.e. Can；
    First, Face datection is carried out to each frame in video, dilatation is carried out to obtained square face mask, increases hair Part, and small rectangle is drawn in two parts respectively, its size and location is determined by the profile size of Face datection；
    Secondly, each pixel in small rectangle is averaged, the average value of three kinds of colors is respectively obtained, as pixel ratio The reference value c of modelb、cg、cr；To realize the effect for automatically giving reference value；The reason is that the pixel of head and face exists Possess specific color data ratio in many scenes, three kinds of colors have certain metastable proportionate relationship, can be applicable in In different scenes, and under different light keep proportionate relationship stabilization；But proportionate relationship is based on reference value cb、cg、cr, And reference value cb、cg、crOtherness is very strong in varying environment difference light；Therefore, the usual important affair of reference value in testing in the past First measure, then carry out the test under identical environment, to realize the dyeing of hair and face；Dyeing refers to that handle meets proportional die The point of type dyes red and purple, represents hair and face respectively；Therefore, the automatic detecting machine system of small rectangle is set, with true Protect in each frame, even if light is different, scene changes occur, also object pixel can be dyed automatically；
    3rd, we have obtained hair and have dyed red, the face of facial purple；We count pixel quantity, The situation of change of each frame is contrasted, if that is, red first increases and then decreases, and change and exceed certain threshold m, then it is judged to completing Elemental motion；If purple first increases and then decreases, and change and exceed certain threshold m, then it is judged to completing once substantially dynamic Make；
    4th, we give changing value δ h to complete once the Hs parameters of the personage of above-mentioned elemental motion；Increase another two at the same time The variable of affecting parameters Hs, one is the Face datection frame i.e. offset of dete frames, for detecting significantly face's displacement, If the situation of change of continuous n frame coordinate offsets meets given rule, and variable quantity reaches the threshold s of setting, then sentences Determine personage and complete an elemental motion；The variable of another affecting parameters Hs is the lasting frame number of the dete frames to disappear, is used for Solve the problems, such as that Face datection frame off and on brings interference；Our solution is：Frame within m frames, which disappears, to be kept Its original continuity, if still continued depletion after m frames, its intensity reduce, gives Hs parameter small change value δ h, presents Go out decline trend；
    Finally, we are by the total variation A of the activity coefficient Hs of each personage of each framer(d) two-dimensional array is stored in, is waited to be calculated When call；
    Step 2: the correlation calculations between two people, draw social networks figure, the corresponding group relation of feature subgraph is excavated；
    1. one active value；
    In video using every k frames as interval of time, d represents the d frames in k frame periods；
    Ar(d)=Σn(δh)
    Its implication is in d frames, and personage completes the changing value added up after the elemental motion of quantity n time, and δ h represent completion every time The corresponding different changing values of different elemental motions, r number for personage, and d represents d frames；
    Ar yArv={ Σ d=1k(Ar(d))}/k
    Its implication is A in y sections of time intervalsr(d) average value of parameter；Since d=1, terminate to d=k, y represents y sections Time interval；R numbers for personage, and d represents d frames；
    2. the correlation calculations between every two people；
    A correlation calculations i.e. Cross Correlation are between each two personage；
    Fτy (l,r)={ Σ d=1(k-t)|(Al y(d)-Al yarv)(Ar y(d+τ)-Ar yarv)|}/(k-τ)(τ>0) when；
    Fτy (l,r)={ Σ d=1(k+t)|(Al y(d-τ)-Al yarv)(Ar y(d)-Ar yarv)|}/(k+τ)(τ<0) when；
    Fτy (l,r)(τ<0)≡Fτy (r,l)(τ>0)
    Fτy (l,r)Implication in y section time intervals, correlation values summation of (l, r) two people when the time difference is τ is put down Average；The positive and negative of wherein τ is used to judge directive property, τ>0 and τ<0 arrow for representing two different directions respectively is directed toward；
    Fmax y (l,r)=max (F τy (l,r)),(-k<τ<k)
    Fmax y (l,r)Represent in the range of the y periods, time difference τ ∈ (- k, k), the corresponding difference F τ of different τ valuesy (l,r)In, The maximum in two people's correlation values sum-total averages is filtered out, and retains its corresponding τ value and is used to judge that arrow is directed toward；
    3. draw social networks figure (referred to as " socialgram ")；
    To the maximal correlation property coefficient Fmax in two human world all in groupy (l,r)It is ranked up, sets selection condition, meeting Setting-out is carried out between all (l, r) two people of condition, is depicted as the socialgram of y sections of time intervals；Socialgram is group relation The basis of excavation；
    4. excavate the corresponding group of feature subgraph；
    Socialgram can be split as ray vertex, triangle, star-like, quadrangle, pentagon feature subgraph；Feature subgraph is introduced such as Under：
    Ray vertex structure, i.e. two rays have same vertices；Corresponding Tn parameters；
    There is line between triangular structure, i.e. three people, form triangle；Corresponding Tr parameters；
    The expansion of star structure, i.e. ray vertex structure, refers to a plurality of ray and meets at a bit；Corresponding Tn parameters；
    Linear structure, represents the line between two people, has generality, appears in each scene；Linear structure includes suitable Gesture straight line connects, and refers to the line between each personage and finally only forms a broken line；Corresponding Lt parameters；
    The corresponding group of each feature subgraph of comprehensive analysis, preparation is provided for the analysis and prediction of group relation；
    Step 3: the analysis and prediction of group relation；
    1. define correlated variables：
    We define Te, Tr, Tn, Lt, Ct, Hs, R, Ji, Jt, 9 variables and are used for ensuing calculating；Wherein Ct is matrix, Other specification is one-dimension array, each personage's numbering l, r in matrix video corresponding with the sequence number in array；
    1) Te parameters reflect speech efficiency, are initially 0, if occur every time it is star-like after limited frame in occur not including me The triangle of thing, increases the variable quantity δ n of personage's Te parameters at this time；Another situation is this personage when statistics occurs star-like every time Outer directional arrow quantity, and increase the variable quantity δ n of corresponding Te values；
    2) Tr parameters reflect participation, are initially 0, and when occurring triangle every time, the Tr values of the appearance personnel in triangle become Change δ n；
    3) Tn parameters correspond to speech number, are initially 0, when occurring star-like every time, the Tn values change δ n of apex personage；
    4) it is not completely independent between the frequent degree and amplitude of Hs parameters respective action, with speech number Tn；Hs=ΣI=1 fAr (d)；Wherein f is video totalframes；Assignment is carried out according to each elemental motion situation is completed in each frame of pretreatment, if reaching Standard, then Hs parameters increase Ar(d) variable quantity；
    5) parameter R=Te/Tn；Meaning directly perceived is the effective ratio of speech；Its effect is similar but not exactly the same with Te；
    6) Ji, Jt are arrow directive property parameter, according to the directive property of every line segment, by Ji, Jt value change δ n of each personage； If arrow is outer direction, Jt Parameters variation δ n, if arrow is interior direction, Ji Parameters variation δ n；It should be noted that Jt Determine Te parameters, when personage meets to appear in the condition on star-like vertex, Te=Te+Jt (d)；Wherein Jt (d) represents d frames Jt values；D represents frame number；
    7) Ct and Lt parameters reflect in group reciprocation degree between two people；Lt parameters represent the line number of two people, if There is line, then numerical value change δ n；Ct matrixes represent the speech level of interaction of two people, its computational methods is to extract continuous star Type sequence, front and rear two personages continuously occurred in sequence are considered as and are once interacted, are stored in the matrix position of reference numeral, such as (l, r), is ranked up matrix numerical value, filters out relation between the strong and weak personage of level of interaction in colony；
    To sum up, summary of parameters.An can be three series by we, be respectively used to judge group interaction degree, dominance, operating frequency and Amplitude；
    2. the analysis method that group relation is excavated；
    First, Ji is analyzed, Jt frequency distribution, excavates the strong personage of prime move in group relation with being inclined to passive personage；Its Secondary, by Tn, Hs frequency distribution, number of talking to each personage is ranked up with there is operating frequency；3rd, analyze Te, Tr, R Frequency distribution, the speech effect to each personage are ranked up with degree of participation；4th, by Lt frequency distribution, two people are closed The power of system is ranked up；Finally, Ct matrix datas are analyzed, talk between personage degree of cooperation, interactive degree are ranked up；
    3. group relation excavates prediction；
    We integrate all the above information and four people's scene group relations in example are sequentially predicted from left to right：
    In terms of group interaction：There is a small amount of talk between No. 1 and No. 2 personages, there is simple talk, and work well, interactive degree is high, matches somebody with somebody Conjunction degree is high；Compared to No. 1 personage of No. 2 personages is closer with No. 3 No. 4 character relations；No. 2 No. 4 personages are more likely in speech State, wherein No. 2 more active, 4 numbers it；No. 3 personage's participations are very high, and speech action effect is high, but liveness of talking is small, energy It is attractive to be inferred as speech content, it is not boring；In addition, the reciprocal in No. 3 No. 4 two human world is also very good, illustrate that two people have There is potential tacit agreement on communication process, relation and topic；
    In terms of dominance：No. 2 No. 4 obvious higher, No. 4 personage's highests；No. 4 personages were both talkative, and speech effect is again high, and participation is again It is good, therefore predict that dominance is stronger compared with No. 2；For No. 2 personages there are certain contradiction, i.e., existing dominance, there is a passive-type；
    In terms of action：No. 1 No. 2 actions are on the high side, it is presumed that No. 3 No. 4 rare headworks, though it can also infer that No. 1 participation is low, But it is not at departing from colony's state；
    Four people's scenes that our trials are surveyed with language reduction：1, No. 2 personage, 2, No. 3 personages, have between 3, No. 4 personages respectively Good communication process, there are closer talk；Overall four relationships are close, and a kind of is the chat field being heightened in spirits between friend Scape, for intimate degree, can determine that most people is friends, higher than strange relationship；Another kind is No. 1 No. 2 and its Other people are antagonistic relations, but are analyzed from status and attitude angle, its prestige is very high, in negotiation scene, and meeting scene；
    In conclusion largely tally with the actual situation in prediction result, in real scene, No. 1 is mother, and No. 2 are daughters, No. 3 It is son, No. 4 are father；Four people are in visits old man in hospital, in corridor chat scenario, wherein talk atmosphere humour, personage Between cooperation also comply with above infer assume；And provide final conspicuousness setting-out；
    Step 4: one manual expression is analyzed with status：
    From group relation is analyzed and predicted, we can further extract one manual expression and character trait；
    1. single stunt expresses；
    First, it is determined that whether personage is active, whether headwork is more；Secondly, the number, whether talkative of analysis personage speech； 3rd, judging personage's speech, whether effect is high, and whether action effect is good, if other people is had an impact, speech plays a role；Most Afterwards, the prime move height of personage is judged, behavior is partially actively or passive；
    In example, No. 1 personage has a small amount of speech generally in the state of listening attentively to；No. 2 personage's speeches are more, in terms of language and action Performance it is more active, the tone is humorous, and content is pertinent and speech of summarizing, in a word can be fascinating；No. 3 personages also have few The amount time is participated in into talk, and is given opinion, but relatively quiet, and liveness is lower slightly；No. 4 personage's talk times are less slightly；
    2. single status prediction：
    No. 4 are even more much-admired personage and elder, are carried weight, strong, and followed by No. 1 personage, strength is strong but less than speech Table；No. 2 No. 3 personages are preceding two people subordinaties, and prestige is lower slightly, the lower personage of posture；Wherein No. 2 are more good at active atmosphere, are said Talk about on the high side, but be not belonging to useless speech, delivering viewpoint and adjusting atmosphere；Other No. 2 personages embody more compared with No. 3 character personalities The personality of contradiction, even more has a strong temperament, and there is the inside and outside mood of polarization；
    To sum up, largely tally with the actual situation in prediction result；In real scene, No. 1 is mother, and No. 2 are daughters, and No. 3 are Son, No. 4 are father；Four people are in visits old man in hospital, between corridor chat scenario, wherein talk atmosphere humour, personage Cooperation also comply with above infer assume；And provide final conspicuousness setting-out；
    Wherein, Face datection, calculates pixel average, graphing, only retains and calculate required information；Walked more than Suddenly, present invention mining data from the more rich video data material of content, can carry out at analysis more extensive video material Reason, is such as applied in friend's chat, discussion, commercial negotiation different scenes；Support to calculate in real time, computer capacity can be converted into Enough data identified and handle；Comprehensive analysis is being carried out more on various dimensions, record group relationship change, solves relation variability Problem, obtains granulating smaller, more deep crowd's interactive relation；So as to solve the reality of more people personage's complex relationships in video More relationships in various scenes are effectively quantified and are evaluated by border problem；The present invention supports following to each actual raw The real-time analysis of scene living, can provide strong method to the character relation analysis in more people's complex scenes and support.
CN201711327006.XA 2017-12-13 2017-12-13 Method for mining social relation of group based on video material Active CN107992598B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201711327006.XA CN107992598B (en) 2017-12-13 2017-12-13 Method for mining social relation of group based on video material

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201711327006.XA CN107992598B (en) 2017-12-13 2017-12-13 Method for mining social relation of group based on video material

Publications (2)

Publication Number Publication Date
CN107992598A true CN107992598A (en) 2018-05-04
CN107992598B CN107992598B (en) 2022-03-15

Family

ID=62037761

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201711327006.XA Active CN107992598B (en) 2017-12-13 2017-12-13 Method for mining social relation of group based on video material

Country Status (1)

Country Link
CN (1) CN107992598B (en)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111428549A (en) * 2019-10-31 2020-07-17 深圳市睿洋图志科技有限公司 Social moving image big data-based character information analysis method and system
CN112433655A (en) * 2020-12-04 2021-03-02 崔秀芬 Information flow interaction processing method based on cloud computing and cloud computing verification interaction center
WO2021120818A1 (en) * 2019-12-20 2021-06-24 Huawei Technologies Co., Ltd. Methods and systems for managing image collection

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20090292549A1 (en) * 2008-05-21 2009-11-26 Honeywell International Inc. Social network construction based on data association
US20100036875A1 (en) * 2008-08-07 2010-02-11 Honeywell International Inc. system for automatic social network construction from image data
US20140270707A1 (en) * 2013-03-15 2014-09-18 Disney Enterprises, Inc. Method and System for Detecting and Recognizing Social Interactions In a Video
CN104778224A (en) * 2015-03-26 2015-07-15 南京邮电大学 Target object social relation identification method based on video semantics

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20090292549A1 (en) * 2008-05-21 2009-11-26 Honeywell International Inc. Social network construction based on data association
US20100036875A1 (en) * 2008-08-07 2010-02-11 Honeywell International Inc. system for automatic social network construction from image data
US20140270707A1 (en) * 2013-03-15 2014-09-18 Disney Enterprises, Inc. Method and System for Detecting and Recognizing Social Interactions In a Video
CN104778224A (en) * 2015-03-26 2015-07-15 南京邮电大学 Target object social relation identification method based on video semantics

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111428549A (en) * 2019-10-31 2020-07-17 深圳市睿洋图志科技有限公司 Social moving image big data-based character information analysis method and system
WO2021120818A1 (en) * 2019-12-20 2021-06-24 Huawei Technologies Co., Ltd. Methods and systems for managing image collection
CN112433655A (en) * 2020-12-04 2021-03-02 崔秀芬 Information flow interaction processing method based on cloud computing and cloud computing verification interaction center

Also Published As

Publication number Publication date
CN107992598B (en) 2022-03-15

Similar Documents

Publication Publication Date Title
Blondel et al. A survey of results on mobile phone datasets analysis
CN104335564B (en) For identify and analyze user personal scene system and method
CN111324643A (en) Knowledge graph generation method, relation mining method, device, equipment and medium
Trier Research note—towards dynamic visualization for understanding evolution of digital communication networks
Savage Class analysis and social research
CN107992598A (en) A kind of method that colony&#39;s social networks excavation is carried out based on video data
CN103916306B (en) The method and apparatus for realizing instant messaging
Lis et al. Where do we want to see other people while relaxing in a city park? Visual relationships with park users and their impact on preferences, safety and privacy
Brantner et al. Constructing public space| New “danger zone” in Europe: Representations of place in social media–supported protests
Davis Social network analysis: An aid in conspiracy investigations
Starnini et al. Robust modeling of human contact networks across different scales and proximity-sensing techniques
DE112019000040T5 (en) DETECTING DETERMINATION MEASURES
Minchev Human factor role for cyber threats resilience
DE202017105869U1 (en) Contextual automatic grouping
CN112699402B (en) Wearable device activity prediction method based on federal personalized random forest
Goldring et al. Transnational Migration and the Reformulation of Analytical Categories: Unpacking Latin American, Refugee Dynamics in Toronto 1
Kadam et al. Social media fake profile detection using data mining technique
Lind et al. Spatio-temporal mobility analysis for community detection in the mobile networks using CDR data
Mann et al. Evolving the world’s systematic creativity methods
Luss et al. Local explanations for reinforcement learning
Waber et al. Understanding organizational behavior with wearable sensing technology
Van et al. Hackernets: Visualizing media conversations on internet of things, big data, and cybersecurity
Beyes ‘The machine could swallow everything’. Satin Island and performing organization
Silva et al. Organizational change and the meaning of time
Gandini et al. Populists or influencers? The use of Facebook videos by populist leaders

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant