CN106503127A - The music data processing method recognized based on facial action and system - Google Patents
The music data processing method recognized based on facial action and system Download PDFInfo
- Publication number
- CN106503127A CN106503127A CN201610912440.3A CN201610912440A CN106503127A CN 106503127 A CN106503127 A CN 106503127A CN 201610912440 A CN201610912440 A CN 201610912440A CN 106503127 A CN106503127 A CN 106503127A
- Authority
- CN
- China
- Prior art keywords
- data
- music
- foreground
- trifle
- facial action
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/60—Information retrieval; Database structures therefor; File system structures therefor of audio data
- G06F16/68—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
- G06F16/683—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
Abstract
The invention provides a kind of music data processing method recognized based on facial action and system, method is：Background music data and foreground music data are obtained, foreground music data are divided into multistage trifle by beat, include multiple bats per trifle；The face action of people is detected within the foreground music time, obtains multiple facial action data, the foreground music data of the corresponding trifle time span of each facial action；Background music data is persistently played, and the foreground music data of a trifle time span corresponding for each facial action data are mated, and combined with background music, generated new music.Music data processing method and system that the present invention is recognized based on facial action, employ based on face's key point technology of identification, the mode that music data and real image are combined, realize interacting for people and corresponding scene, need not be aided in by outside setting, implementation is simple, improves Consumer's Experience.
Description
Technical field
The present invention relates to data processing field, more particularly to the music data process recognized based on facial action.
Background technology
In prior art, it is (such as Wii, dancing machine using mouse, keyboard, joystick, Touch Screen, external sensor mostly
Pedal etc.), the mode such as human posture (such as Kinect) operate, realize interacting for people and corresponding scene.Wherein, human posture's operation
Mode, needs the information such as deep space, it is necessary to using specific device；It is thus desirable to detecting whole body, needs larger space ability
Game.In addition there is the problems such as accuracy rate is low, side-play amount is high, cause the interesting reduction in interaction；In addition such shape
Scene interactivity is most to detect (example for human action：Raise one's hand, skirt), less combined with real image.
The application of face's key point identification at present is mainly combined with image composing technique, and face is become other animals, worn
Upper different ornaments simultaneously combine facial action generation animation etc..
Thinning face is provided with the application during facial movements such as face's rehabilitation：Traditional facial movement through word flow process description or
Reach the effect of facial movement through the face motion demonstration in study film, but such facial movement process not with true
Image is combined, and Experience Degree is low.
Therefore, defect of the prior art is, for interacting for people and corresponding scene, needs real by external equipment auxiliary
Existing, implementation is single, and Consumer's Experience is low.Face's key point technology of identification and real image cannot be combined.
Content of the invention
For above-mentioned technical problem, the present invention is provided a kind of music data processing method recognized based on facial action and is
System, employ based on face's key point technology of identification, the mode that music data and real image are combined, realize people with corresponding
The interaction of scene, it is not necessary to aided in by outside setting, implementation is simple, improves Consumer's Experience.
For solving above-mentioned technical problem, the technical scheme that the present invention is provided is：
In a first aspect, the present invention provides a kind of music data processing method recognized based on facial action, including：
Step S1, obtains background music data and foreground music data, the background music data and the foreground music
Data are respectively the music of a hop count second to several minutes length；
The foreground music data are divided into multistage trifle by beat by step S2, include multiple bats per trifle；
Step S3, detects the face action of people within the foreground music time, and acquisition is more within the foreground music time
Individual facial action data, the foreground music data of the corresponding trifle time span of each facial action；
Step S4, the background music data is persistently played, and will be right with which for each facial action data described
The foreground music data of the trifle time span that answers are mated, and are combined with the background music, generate new music.
The technical scheme is that：Background music data and foreground music data, the background music data are first obtained
The music of a hop count second to several minutes length is respectively with the foreground music data；Then by the foreground music data by section
Bat is divided into multistage trifle, includes multiple bats per trifle；
Then the face action of people, acquisition multiple faces during foreground music time are detected within the foreground music time
Portion's action data, the foreground music data of the corresponding trifle time span of each facial action；Finally by the background music number
According to persistently being played, and the foreground music by the trifle time span corresponding for each facial action data described
Data are mated, and are combined with the background music, generate new music.
The music data processing method that the present invention is recognized based on facial action, employs based on face's key point identification skill
Art, the mode that music data and real image are combined realize interacting for people and corresponding scene, it is not necessary to by outside setting
Aided in, implementation is simple, improves Consumer's Experience.
Further, after step S2, also include：
The background music data is persistently played, target face action data, target face action data is obtained
Corresponding unique trifle foreground music；
According to the target face action data, obtain and move in the face started per trifle in before and after's bit time
Make data；
The facial action data are carried out mating judgement with the target face action data, a trifle foreground voice is carried out
The happy selection that plays：
When the facial action data are mated with the target face action data, the trifle foreground music is played,
The one trifle foreground music is and the unique corresponding foreground music of the target face action data；
When the facial action data are mismatched with the target face action data, the facial action data are not played
Corresponding trifle foreground music.
Further, after step S2, also include：
The background music data is persistently played, obtain virtual scene data, the virtual scene data be to
The virtual scene data of each position mobile object in human face；
According to the virtual scene data of each position mobile object in human face, corresponding facial action number is obtained
According to the acquisition of the corresponding facial action data is before the position that the mobile object reaches in the human face；
The facial action data are carried out mating judgement with the virtual scene data, the virtual scene data are carried out
In corresponding mobile object process：
When the facial action data and the virtual scene Data Matching, will correspondingly move in the virtual scene data
Object is removed；
When the facial action data are mismatched with the virtual scene data, to correspondingly moving in the virtual scene data
Animal body does not deal with；
After the rear bit time started per trifle, without corresponding facial action and target face action data
Match somebody with somebody, corresponding mobile object in the virtual scene data is removed.
Further, will after in the virtual scene data, corresponding mobile object is removed, including：
Obtain the effect data that corresponding mobile object is removed in the virtual scene data；
According to the effect data that corresponding mobile object in the virtual scene data is removed, to the virtual scene data
The facial action data of coupling are evaluated, and obtain evaluation result.
Further, the identification of facial action is carried out by the identification of face key point and fuzzy control theory.
A kind of music data processing method recognized based on facial action of the present invention, based on face's key point technology of identification,
The mode that music data and real image are combined, realizes interacting for people and corresponding scene, i.e., the knowledge being failure to actuate by face
Not, mated with the music data of corresponding scene, realized the creation of music, corresponding virtual objects in the broadcasting of music and scene
Elimination, be presented in user in animated way at the moment, it is not necessary to aided in by outside setting, implementation is simple, carries
High Consumer's Experience.
Further, also include：
The background music data is persistently played, target face action data, the target face action is obtained
The corresponding trifle foreground music of data, the foreground music is divided into the first foreground music and the second foreground music, before described first
Scape music mates broadcasting with the background music, and second foreground music is mismatched with the background music and played；
According to the target face action data, obtain and move in the face started per trifle in before and after's bit time
Make data；
The facial action data are carried out mating judgement with the target face action data, the target face is carried out
The broadcasting of the corresponding trifle foreground music of action data is chosen：
When the facial action data are mated with the target face action data, the target face action data is played
A corresponding trifle foreground music, and the trifle foreground music is the first prospect corresponding with the target face action data
Music；
When the facial action data are mismatched with the target face action data, the target face action number is played
According to a corresponding trifle foreground music, and the trifle foreground music be with before the target face action data corresponding second
Scape music.
After facial action data mate target face action data successful match with subject performance data, corresponding broadcasting should
The corresponding trifle foreground music of target face action, also, the first foreground music and background music match the broadcasting of harmony,
, whereas if the facial action data that user makes are mismatched with target face action data, then correspondingly play target face and move
Make corresponding second foreground music, the second foreground music does not match that harmonious broadcasting with background music, so by playing
The difference of music can judge facial action data that user makes, and whether the match is successful with target face action data, improves
Consumer's Experience.
Second aspect, the invention provides a kind of music data processing system recognized based on facial action, including：
Music data acquisition module, for obtaining background music data and foreground music data, the background music data
The music of a hop count second to several minutes length is respectively with the foreground music data；
Music data processing module, for the foreground music data are divided into multistage trifle by beat, includes per trifle
Multiple bats；
Facial action acquisition module, for detecting the face action of people within the foreground music time, obtains in the prospect
Multiple facial action data in musical time, the foreground music data of the corresponding trifle time span of each facial action；
Musical composition module, for persistently being played the background music data, and by each facial action described
The foreground music data of the corresponding trifle time span of data are mated, and are combined with the background music, raw
The music of Cheng Xin.
The technical scheme is that：Music data acquisition module is first passed through, background music data and foreground music is obtained
Data, the background music data and the foreground music data are respectively the music of a hop count second to several minutes length；Then
By music data processing module, the foreground music data are divided into multistage trifle by beat, include multiple bats per trifle；
Facial action acquisition module is then passed through, and the face action of people is detected within the foreground music time, is obtained described
Multiple facial action data in the foreground music time, the foreground music number of the corresponding trifle time span of each facial action
According to；Finally by musical composition module, the background music data is persistently played, and by each facial action number described
Mated according to the foreground music data of the corresponding trifle time span, combined with the background music, generated
New music.
A kind of music data processing system recognized based on facial action of the present invention, is employed based on face's key point identification
Technology, the mode that music data and real image are combined realize interacting for people and corresponding scene, it is not necessary to set by outside
Put and aided in, implementation is simple, improves Consumer's Experience.
Further, after the music data processing module, also include the selection of music playing module, be used for：
The background music data is persistently played, target face action data, target face action data is obtained
Corresponding unique trifle foreground music；
According to the target face action data, obtain and move in the face started per trifle in before and after's bit time
Make data；
The facial action data are carried out mating judgement with the target face action data, a trifle foreground voice is carried out
The happy selection that plays：
When the facial action data are mated with the target face action data, the trifle foreground music is played,
The one trifle foreground music is and the unique corresponding foreground music of the target face action data；
When the facial action data are mismatched with the target face action data, the facial action data are not played
Corresponding trifle foreground music.
Further, after the music data processing module, also include music virtual scene module, be used for：
The background music data is persistently played, obtain virtual scene data, the virtual scene data be to
The virtual scene data of each position mobile object in human face；
According to the virtual scene data of each position mobile object in human face, corresponding facial action number is obtained
According to the acquisition of the corresponding facial action data is before the position that the mobile object reaches in the human face；
The facial action data are carried out mating judgement with the virtual scene data, the virtual scene data are carried out
In corresponding mobile object process：
When the facial action data and the virtual scene Data Matching, will correspondingly move in the virtual scene data
Object is removed；
When the facial action data are mismatched with the virtual scene data, to correspondingly moving in the virtual scene data
Animal body does not deal with；
After the rear bit time started per trifle, without corresponding facial action and target face action data
Match somebody with somebody, corresponding mobile object in the virtual scene data is removed.
Further, in the music virtual scene module, including effect assessment submodule, by the virtual scene number
After according in, corresponding mobile object is removed, the effect assessment submodule is used for：
Obtain the effect data that corresponding mobile object is removed in the virtual scene data；
According to the effect data that corresponding mobile object in the virtual scene data is removed, to the virtual scene data
The facial action data of coupling are evaluated, and obtain evaluation result.
Description of the drawings
In order to be illustrated more clearly that the specific embodiment of the invention or technical scheme of the prior art, below will be to concrete
Needed for embodiment or description of the prior art, accompanying drawing to be used is briefly described.
Fig. 1 shows a kind of music data process side recognized based on facial action that first embodiment of the invention is provided
The flow chart of method；
Fig. 2 shows a kind of music data process side recognized based on facial action that first embodiment of the invention is provided
First schematic diagram of time shafts in method；
Fig. 3 shows a kind of music data process side recognized based on facial action that first embodiment of the invention is provided
Second schematic diagram of time shafts in method；
Fig. 4 shows a kind of music data processing system recognized based on facial action that second embodiment of the invention is provided
The schematic diagram of system.
Specific embodiment
The embodiment of technical solution of the present invention is described in detail below in conjunction with accompanying drawing.Following examples are only used for
Technical scheme is described clearly, is therefore intended only as example, and the protection of the present invention can not be limited with this
Scope.
Embodiment one
Fig. 1 shows a kind of music data process side recognized based on facial action that first embodiment of the invention is provided
The flow chart of method；As shown in figure 1, the embodiment of the present invention one provides a kind of music data process side recognized based on facial action
Method, including：
Step S1, obtains background music data and foreground music data, background music data and foreground music data difference
Music for a hop count second to several minutes length；
Foreground music data are divided into multistage trifle by beat by step S2, include multiple bats per trifle；
Step S3, detects the face action of people, multiple faces of the acquisition within the foreground music time within the foreground music time
Portion's action data, the foreground music data of the corresponding trifle time span of each facial action；
Step S4, background music data is persistently played, and little by corresponding for each facial action data one
The foreground music data of section time span are mated, and are combined with background music, generate new music.
The technical scheme is that：Background music data and foreground music data are first obtained, background music data and front
Scape music data is respectively the music of a hop count second to several minutes length；Then foreground music data are divided into multistage by beat little
Section, includes multiple bats per trifle；
Then the face action of people is detected within the foreground music time, and multiple faces of the acquisition within the foreground music time are moved
Make data, the foreground music data of the corresponding trifle time span of each facial action；Finally background music data is held
Continued broadcasting is put, and the foreground music data of a trifle time span corresponding for each facial action data are mated, with
Background music is combined, and generates new music.
The music data processing method that the present invention is recognized based on facial action, can arrange different scene informations, different
Scene information have different demands, need to complete according to different demands in scene information at the appointed time, if different
Scene information mate from different facial actions, then can realize different actions, and then realize the interaction of various people and scene.
The music data processing method that the present invention is recognized based on facial action, employs based on face's key point identification skill
Art, the mode that music data and real image are combined realize interacting for people and corresponding scene, it is not necessary to by outside setting
Aided in, implementation is simple, improves Consumer's Experience.
Specifically, after step S2, also include：
Background music data is persistently played, target face action data is obtained, target face action data is corresponding
Unique trifle foreground music；
According to target face action data, obtain and start the facial action data in before and after's bit time in every trifle；
Facial action data and target face action data are carried out mating judgement, trifle foreground music broadcasting is carried out
Choose：
When facial action data are mated with target face action data, a trifle foreground music, a trifle foreground voice is played
Pleasure is and the unique corresponding foreground music of target face action data；
When facial action data are mismatched with target face action data, the corresponding trifle of facial action data is not played
Foreground music.
Specifically, the foreground music of many trifles is set, and per trifle foreground music, corresponding only one facial action, corresponding
After the match is successful in first facial action and first aim facial action, next facial action is carried out with next target
The coupling of facial action, so can realize the continuous broadcasting of foreground music according to different target face actions.
Specifically, after step S2, also include：
Background music data is persistently played, obtain virtual scene data, virtual scene data be to human face in
The virtual scene data of each position mobile object；
According to the virtual scene data of each position mobile object in human face, corresponding facial action data are obtained,
The acquisition of corresponding facial action data is before the position that mobile object reaches in human face；
Facial action data and virtual scene data are carried out mating judgement, corresponding motive objects in virtual scene data are carried out
The process of body：
When facial action data and virtual scene Data Matching, corresponding mobile object in virtual scene data is removed；
When facial action data are mismatched with virtual scene data, corresponding mobile object in virtual scene data is not made to locate
Reason；
After the rear bit time that every trifle starts, mate with target face action data without corresponding facial action,
Corresponding mobile object in virtual scene data is removed.
In conjunction with AR technology, in real scene, corresponding face facial positions arrange different virtual objects and move to face,
Before and after the time point that virtual objects are moved in face facial positions, bit time is event horizon, correspondingly makes each virtual
The corresponding elimination action of article, the facial action done by people and elimination action are then done next action, are increased after the match is successful
Interest, while the match is successful or unsuccessfully has different audios and animation.Through after a period of time, match time shortens,
Coupling eliminates speed of action and can accelerate.
Specifically, will after in virtual scene data, corresponding mobile object is removed, including：
Obtain the effect data that corresponding mobile object is removed in virtual scene data；
According to the effect data that corresponding mobile object in virtual scene data is removed, to the face with virtual scene Data Matching
Portion's action data is evaluated, and obtains evaluation result.
As shown in Fig. 2 transverse axis is time shafts, time, the right represent later time earlier for left side representative.Relatively thick length
The longitudinal axis is trifle separation, and the separation of point is clapped in shorter representative.In this legend, it is before every trifle starts to drive scope
Two bats are all correct driving, and 0.5 bat before and after starting previous bat with every trifle is that perfection drives, in such as Fig. 2, the part that e shows, and e
Before and after 0.5 clap as common driving, the part that such as f shows.Perfect, common driving time and scope all freely can be replaced.With
The method is doing the evaluation criterion to virtual scene Data Matching facial action data.
Specifically, the identification of facial action is carried out by the identification of face key point and fuzzy control theory.
In the present invention, processed based on above-mentioned music data, be based on the identification of face key point, set up a reliability
Face system, finally accurately recognizes facial action with reference to fuzzy control theory：Blink, cross-eye, choose eyebrow, frown, the nose that wrinkles,
Loll, mouth of beeping, open one's mouth, wapperijaw, lick lip, close lightly lip, nod, head left rotation and right rotation, head rotate etc. up and down.Wherein face is crucial
Technology of identification is put for known prior art, here does not do excessive narration.
Specifically, also include：
Background music data is persistently played, target face action data is obtained, target face action data is corresponding
One trifle foreground music, foreground music are divided into the first foreground music and the second foreground music, the first foreground music and background music
Coupling is played, and the second foreground music is mismatched with background music and played；
According to target face action data, obtain and start the facial action data in before and after's bit time in every trifle；
Facial action data and target face action data are carried out mating judgement, target face action data is carried out corresponding
The broadcasting of one trifle foreground music is chosen：
When facial action data are mated with target face action data, before playing the corresponding trifle of target face action data
Scape music, and a trifle foreground music is the first foreground music corresponding with target face action data；
When facial action data are mismatched with target face action data, the corresponding trifle of target face action data is played
Foreground music, and a trifle foreground music is the second foreground music corresponding with target face action data.
After facial action data mate target face action data successful match with subject performance data, corresponding broadcasting should
The corresponding trifle foreground music of target face action, also, the first foreground music and background music match the broadcasting of harmony,
, whereas if the facial action data that user makes are mismatched with target face action data, then correspondingly play target face and move
Make corresponding second foreground music, the second foreground music does not match that harmonious broadcasting with background music, so by playing
The difference of music can judge facial action data that user makes, and whether the match is successful with target face action data, improves
Consumer's Experience.
Specifically, the first foreground music may be configured as representing that successful music, the second foreground music can be set to represent and lose
The foreground music for losing, so makes music more having any different property.
As shown in figure 3, transverse axis is time shafts, time, the right represent later time earlier for left side representative.Relatively thick length
The longitudinal axis is trifle separation, and the separation of point is clapped in shorter representative.Example in Fig. 3, the scope of c is a trifle, the model of d
Enclose for a bat.A for detecting time point, represent in Fig. 3 every trifle start before and after clap for detect time point, any at this moment
Between put the facial action that does and can be detected.B is sphere of action, generally in units of a trifle, represents here in this figure little
Do in the range of a bat before and after section and express one's feelings, can all drive this trifle to make feedback.Per a few bats of trifle, detect time point and effect model
Enclose and all freely can replace.
A kind of music data processing method recognized based on facial action of the present invention, based on face's key point technology of identification,
The mode that music data and real image are combined, realizes interacting for people and corresponding scene, i.e., the knowledge being failure to actuate by face
Not, mated with the music data of corresponding scene, realized the creation of music, corresponding virtual objects in the broadcasting of music and scene
Elimination, be presented in user in animated way at the moment, it is not necessary to aided in by outside setting, implementation is simple, carries
High Consumer's Experience.
Embodiment two
Fig. 4 shows a kind of music data processing system recognized based on facial action that second embodiment of the invention is provided
The schematic diagram of system；As shown in figure 4, the embodiment of the present invention two provides a kind of music data processing system recognized based on facial action
System 10, including：
Music data acquisition module 101, for obtaining background music data and foreground music data, background music data and
Foreground music data are respectively the music of a hop count second to several minutes length；
Music data processing module 102, for foreground music data are divided into multistage trifle by beat, per trifle comprising many
Individual bat；
Facial action acquisition module 103, for detecting the face action of people within the foreground music time, obtains in foreground voice
Multiple facial action data in the happy time, the foreground music data of the corresponding trifle time span of each facial action；
Musical composition module 104, for background music data is persistently played, and by each facial action data with
The foreground music data of its corresponding trifle time span are mated, and are combined with background music, generate new music.
The technical scheme is that：Music data acquisition module 101 is first passed through, background music data and foreground voice is obtained
Happy data, background music data and foreground music data are respectively the music of a hop count second to several minutes length；Then pass through sound
Foreground music data are divided into multistage trifle by beat by happy data processing module 102, include multiple bats per trifle；
Facial action acquisition module 103 is then passed through, and the face action of people is detected within the foreground music time, is obtained front
Multiple facial action data in scape musical time, the foreground music data of the corresponding trifle time span of each facial action；
Finally by musical composition module 104, background music data is persistently played, and will be right with which for each facial action data
The foreground music data of the trifle time span that answers are mated, and are combined with background music, generate new music.
A kind of music data processing system 10 recognized based on facial action of the present invention, is employed and is known based on face's key point
Other technology, the mode that music data and real image are combined realize interacting for people and corresponding scene, it is not necessary to by outside
Setting is aided in, and implementation is simple, improves Consumer's Experience.
Specifically, after music data processing module 102, also include the selection of music playing module, be used for：
Background music data is persistently played, target face action data is obtained, target face action data is corresponding
Unique trifle foreground music；
According to target face action data, obtain and start the facial action data in before and after's bit time in every trifle；
Facial action data and target face action data are carried out mating judgement, trifle foreground music broadcasting is carried out
Choose：
When facial action data are mated with target face action data, a trifle foreground music, a trifle foreground voice is played
Pleasure is and the unique corresponding foreground music of target face action data；
When facial action data are mismatched with target face action data, the corresponding trifle of facial action data is not played
Foreground music.
Specifically, after music data processing module 102, also include music virtual scene module, be used for：
Background music data is persistently played, obtain virtual scene data, virtual scene data be to human face in
The virtual scene data of each position mobile object；
According to the virtual scene data of each position mobile object in human face, corresponding facial action data are obtained,
The acquisition of corresponding facial action data is before the position that mobile object reaches in human face；
Facial action data and virtual scene data are carried out mating judgement, corresponding motive objects in virtual scene data are carried out
The process of body：
When facial action data and virtual scene Data Matching, corresponding mobile object in virtual scene data is removed；
When facial action data are mismatched with virtual scene data, corresponding mobile object in virtual scene data is not made to locate
Reason；
After the rear bit time that every trifle starts, mate with target face action data without corresponding facial action,
Corresponding mobile object in virtual scene data is removed.
Specifically, in music virtual scene module 104, including effect assessment submodule, will be right in virtual scene data
After answering mobile object to remove, effect assessment submodule is used for：
Obtain the effect data that corresponding mobile object is removed in virtual scene data；
According to the effect data that corresponding mobile object in virtual scene data is removed, to the face with virtual scene Data Matching
Portion's action data is evaluated, and obtains evaluation result.
Specifically, also include facial action identification module 100, for recognizing by face key point and fuzzy control theory
Carry out the identification of facial action.
Specifically, also include the selection of music playing module, be used for：
Background music data is persistently played, target face action data is obtained, target face action data is corresponding
One trifle foreground music, foreground music are divided into the first foreground music and the second foreground music, the first foreground music and background music
Coupling is played, and the second foreground music is mismatched with background music and played；
According to target face action data, obtain and start the facial action data in before and after's bit time in every trifle；
Facial action data and target face action data are carried out mating judgement, target face action data is carried out corresponding
The broadcasting of one trifle foreground music is chosen：
When facial action data are mated with target face action data, before playing the corresponding trifle of target face action data
Scape music, and a trifle foreground music is the first foreground music corresponding with target face action data；
When facial action data are mismatched with target face action data, the corresponding trifle of target face action data is played
Foreground music, and a trifle foreground music is the second foreground music corresponding with target face action data.
After facial action data mate target face action data successful match with subject performance data, corresponding broadcasting should
The corresponding trifle foreground music of target face action, also, the first foreground music and background music match the broadcasting of harmony,
, whereas if the facial action data that user makes are mismatched with target face action data, then correspondingly play target face and move
Make corresponding second foreground music, the second foreground music does not match that harmonious broadcasting with background music, so by playing
The difference of music can judge facial action data that user makes, and whether the match is successful with target face action data, improves
Consumer's Experience.
Specifically, the first foreground music may be configured as representing that successful music, the second foreground music can be set to represent and lose
The foreground music for losing, so makes music more having any different property.
A kind of music data processing system recognized based on facial action of the present invention, based on face's key point technology of identification,
The mode that music data and real image are combined, realizes interacting for people and corresponding scene, i.e., by the knowledge of face action
Not, mated with the music data of corresponding scene, realized the creation of music, corresponding virtual objects in the broadcasting of music and scene
Elimination, be presented in user in animated way at the moment, it is not necessary to aided in by outside setting, implementation is simple, carries
High Consumer's Experience.
Embodiment three
A kind of music data processing method recognized based on facial action in conjunction with the embodiment of the present invention one, and embodiment
A kind of music data processing system recognized based on facial action in two, is illustrated in conjunction with specific scene of game.
Scene one
Melody is created：Before background music is persistently played, and each facial action corresponds to one section of one trifle time span
Scape music, in the facial action that every trifle starts to detect in before and after's bit time, can all drive the trifle to play corresponding
Foreground music.A series of facial actions are such as set, blink, choose eyebrow, frown, the nose that wrinkles etc., then recognize the facial action of people, root
According to the difference of the facial action of identification, the different music of corresponding broadcasting, the trifle music that plays every time constitute different music,
User is made to do different music according to the mood of oneself wound.
Scene two
Music game：Background music and foreground music are the music of a hop count minute, and background music is persistently played, play according to
So in units of trifle, specified facial action is randomly generated by game per little festival-gathering, player must be in driving time (per trifle
Bit time before and after beginning) complete to specify facial action, the foreground music of the trifle persistently can play；If there is more than one prospect
Music, also may done correct expression more multiple after by music be superimposed up.Motivational music is combined with human face action, trip is made
Play is challenging and interesting.
Scene three
AR Rhythmic games：Background music is persistently played, and having different articles on picture has past each portion on the face of rhythm
Displacement is moved, and player must do corresponding expression and remove the article when article moves to the position.For example：Mosquito flies toward eyes
Past, it is necessary to which it is killed by blink when mosquito reaches eyes.Success (be divided into perfection, common) unsuccessfully has different sounds
Effect, animation and score.Through after a period of time, rhythm tempo can be accelerated, increase game degree of difficulty.
The image for directly being obtained using photographic head is made face's action recognition to operate game, and can in real time with true face's shadow
Interesting animation effect is produced as combining.This mode of operation is not required to other purchase of equipment, you can differentiates face in real time, exactly and moves
Make；And do not need extremity mode of operation, can also benefit handicapped people, allow them also enjoy the enjoyment of game.
Finally it should be noted that：Various embodiments above only in order to technical scheme to be described, rather than a limitation；To the greatest extent
Pipe has been described in detail to the present invention with reference to foregoing embodiments, it will be understood by those within the art that：Its according to
So the technical scheme described in foregoing embodiments can be modified, or which part or all technical characteristic are entered
Row equivalent；And these modifications or replacement, do not make the essence of appropriate technical solution depart from various embodiments of the present invention technology
The scope of scheme, its all should cover in the middle of the scope of the claim and description of the present invention.
Claims (10)
1. the music data processing method for being recognized based on facial action, it is characterised in that include：
Step S1, obtains background music data and foreground music data, the background music data and the foreground music data
The music of a respectively hop count second to several minutes length；
The foreground music data are divided into multistage trifle by beat by step S2, include multiple bats per trifle；
Step S3, detects the face action of people, multiple faces of the acquisition within the foreground music time within the foreground music time
Portion's action data, the foreground music data of the corresponding trifle time span of each facial action；
Step S4, the background music data is persistently played, and will be corresponding for each facial action data described
The foreground music data of the one trifle time span are mated, and are combined with the background music, generate new music.
2. the music data processing method for being recognized based on facial action according to claim 1, it is characterised in that the step
After S2, also include：
The background music data is persistently played, target face action data is obtained, target face action data is corresponding
Unique trifle foreground music；
According to the target face action data, obtain and start the facial action number in before and after's bit time described per trifle
According to；
The facial action data are carried out mating judgement with the target face action data, a trifle foreground music is carried out and is broadcast
The selection that puts：
When the facial action data are mated with the target face action data, the trifle foreground music is played, described
One trifle foreground music is and the unique corresponding foreground music of the target face action data；
When the facial action data are mismatched with the target face action data, the facial action data are not played corresponding
A trifle foreground music.
3. the music data processing method for being recognized based on facial action according to claim 1, it is characterised in that
After step S2, also include：
The background music data is persistently played, virtual scene data are obtained, the virtual scene data are to people face
The virtual scene data of each position mobile object in portion；
According to the virtual scene data of each position mobile object in human face, corresponding facial action data are obtained,
The acquisition of the corresponding facial action data is before the position that the mobile object reaches in the human face；
The facial action data are carried out mating judgement with the virtual scene data, it is right in the virtual scene data to carry out
Answer the process of mobile object：
When the facial action data and the virtual scene Data Matching, by corresponding mobile object in the virtual scene data
Remove；
When the facial action data are mismatched with the virtual scene data, to corresponding motive objects in the virtual scene data
Body does not deal with；
After the rear bit time started per trifle, mate with target face action data without corresponding facial action,
Corresponding mobile object in the virtual scene data is removed.
4. the music data processing method for being recognized based on facial action according to claim 3, it is characterised in that
After in the virtual scene data, corresponding mobile object is removed, including：
Obtain the effect data that corresponding mobile object is removed in the virtual scene data；
According to the effect data that corresponding mobile object in the virtual scene data is removed, to the virtual scene Data Matching
The facial action data evaluated, obtain evaluation result.
5. the music data processing method for being recognized based on facial action according to claim 1, it is characterised in that
The identification that facial action is carried out by the identification of face key point and fuzzy control theory.
6. according to claim 1 or claim 2 based on facial action recognize music data processing method, it is characterised in that
Also include：
The background music data is persistently played, target face action data, the target face action data is obtained
A corresponding trifle foreground music, the foreground music are divided into the first foreground music and the second foreground music, first foreground voice
Pleasure mates broadcasting with the background music, and second foreground music is mismatched with the background music and played；
According to the target face action data, obtain and start the facial action number in before and after's bit time described per trifle
According to；
The facial action data are carried out mating judgement with the target face action data, the target face action is carried out
The broadcasting of the corresponding trifle foreground music of data is chosen：
When the facial action data are mated with the target face action data, the target face action data is played corresponding
One trifle foreground music, and the trifle foreground music is the first foreground voice corresponding with the target face action data
Happy；
When the facial action data are mismatched with the target face action data, the target face action data pair is played
A trifle foreground music is answered, and the trifle foreground music is the second foreground voice corresponding with the target face action data
Happy.
7. the music data processing system for being recognized based on facial action, it is characterised in that include：
Music data acquisition module, for obtaining background music data and foreground music data, the background music data and institute
State the music that foreground music data are respectively a hop count second to several minutes length；
Music data processing module, for the foreground music data are divided into multistage trifle by beat, per trifle comprising multiple
Bat；
Facial action acquisition module, for detecting the face action of people within the foreground music time, obtains in the foreground music
Multiple facial action data in time, the foreground music data of the corresponding trifle time span of each facial action；
Musical composition module, for persistently being played the background music data, and by each facial action data described
The foreground music data of the corresponding trifle time span are mated, and are combined with the background music, generate new
Music.
8. the music data processing system for being recognized based on facial action according to claim 7, it is characterised in that
After the music data processing module, also include the selection of music playing module, be used for：
The background music data is persistently played, target face action data is obtained, target face action data is corresponding
Unique trifle foreground music；
According to the target face action data, obtain and start the facial action number in before and after's bit time described per trifle
According to；
The facial action data are carried out mating judgement with the target face action data, a trifle foreground music is carried out and is broadcast
The selection that puts：
When the facial action data are mated with the target face action data, the trifle foreground music is played, described
One trifle foreground music is and the unique corresponding foreground music of the target face action data；
When the facial action data are mismatched with the target face action data, the facial action data are not played corresponding
A trifle foreground music.
9. the music data processing system for being recognized based on facial action according to claim 7, it is characterised in that
After the music data processing module, also include music virtual scene module, be used for：
The background music data is persistently played, virtual scene data are obtained, the virtual scene data are to people face
The virtual scene data of each position mobile object in portion；
According to the virtual scene data of each position mobile object in human face, corresponding facial action data are obtained,
The acquisition of the corresponding facial action data is before the position that the mobile object reaches in the human face；
The facial action data are carried out mating judgement with the virtual scene data, it is right in the virtual scene data to carry out
Answer the process of mobile object：
When the facial action data and the virtual scene Data Matching, by corresponding mobile object in the virtual scene data
Remove；
When the facial action data are mismatched with the virtual scene data, to corresponding motive objects in the virtual scene data
Body does not deal with；
After the rear bit time started per trifle, mate with target face action data without corresponding facial action,
Corresponding mobile object in the virtual scene data is removed.
10. the music data processing system for being recognized based on facial action according to claim 9, it is characterised in that
In the music virtual scene module, including effect assessment submodule, will correspondingly move in the virtual scene data
After object is removed, the effect assessment submodule is used for：
Obtain the effect data that corresponding mobile object is removed in the virtual scene data；
According to the effect data that corresponding mobile object in the virtual scene data is removed, to the virtual scene Data Matching
The facial action data evaluated, obtain evaluation result.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610912440.3A CN106503127B (en) | 2016-10-19 | 2016-10-19 | Music data processing method and system based on facial action identification |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610912440.3A CN106503127B (en) | 2016-10-19 | 2016-10-19 | Music data processing method and system based on facial action identification |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106503127A true CN106503127A (en) | 2017-03-15 |
| CN106503127B CN106503127B (en) | 2019-09-27 |
Family
ID=58294244
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610912440.3A Active CN106503127B (en) | 2016-10-19 | 2016-10-19 | Music data processing method and system based on facial action identification |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106503127B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108905193A (en) * | 2018-07-03 | 2018-11-30 | 百度在线网络技术（北京）有限公司 | Game manipulates processing method, equipment and storage medium |
| CN109343770A (en) * | 2018-09-27 | 2019-02-15 | 腾讯科技（深圳）有限公司 | Interaction feedback method, equipment and recording medium |
| CN110047520A (en) * | 2019-03-19 | 2019-07-23 | 北京字节跳动网络技术有限公司 | Control method, device, electronic equipment and the computer readable storage medium that audio plays |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1764940A (en) * | 2003-03-31 | 2006-04-26 | 索尼株式会社 | Tempo analysis device and tempo analysis method |
| CN101836219A (en) * | 2007-11-01 | 2010-09-15 | 索尼爱立信移动通讯有限公司 | Generating music playlist based on facial expression |
| CN102640149A (en) * | 2009-12-04 | 2012-08-15 | 索尼计算机娱乐公司 | Music recommendation system, information processing device, and information processing method |
| CN102880388A (en) * | 2012-09-06 | 2013-01-16 | 北京天宇朗通通信设备股份有限公司 | Music processing method, music processing device and mobile terminal |
| CN102929476A (en) * | 2012-09-06 | 2013-02-13 | 北京天宇朗通通信设备股份有限公司 | Method and device for controlling main menu of terminal |
| CN103383694A (en) * | 2012-12-14 | 2013-11-06 | 李博文 | Method and system for organizing, managing and marking music document |
| CN104851435A (en) * | 2015-06-06 | 2015-08-19 | 孔霞 | Music intelligent playing method based on network communication |
| CN105518783A (en) * | 2013-08-19 | 2016-04-20 | 谷歌公司 | Content-based video segmentation |
-
2016
- 2016-10-19 CN CN201610912440.3A patent/CN106503127B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1764940A (en) * | 2003-03-31 | 2006-04-26 | 索尼株式会社 | Tempo analysis device and tempo analysis method |
| CN101836219A (en) * | 2007-11-01 | 2010-09-15 | 索尼爱立信移动通讯有限公司 | Generating music playlist based on facial expression |
| CN102640149A (en) * | 2009-12-04 | 2012-08-15 | 索尼计算机娱乐公司 | Music recommendation system, information processing device, and information processing method |
| CN102880388A (en) * | 2012-09-06 | 2013-01-16 | 北京天宇朗通通信设备股份有限公司 | Music processing method, music processing device and mobile terminal |
| CN102929476A (en) * | 2012-09-06 | 2013-02-13 | 北京天宇朗通通信设备股份有限公司 | Method and device for controlling main menu of terminal |
| CN103383694A (en) * | 2012-12-14 | 2013-11-06 | 李博文 | Method and system for organizing, managing and marking music document |
| CN105518783A (en) * | 2013-08-19 | 2016-04-20 | 谷歌公司 | Content-based video segmentation |
| CN104851435A (en) * | 2015-06-06 | 2015-08-19 | 孔霞 | Music intelligent playing method based on network communication |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108905193A (en) * | 2018-07-03 | 2018-11-30 | 百度在线网络技术（北京）有限公司 | Game manipulates processing method, equipment and storage medium |
| CN108905193B (en) * | 2018-07-03 | 2022-04-15 | 百度在线网络技术（北京）有限公司 | Game manipulation processing method, device and storage medium |
| CN109343770A (en) * | 2018-09-27 | 2019-02-15 | 腾讯科技（深圳）有限公司 | Interaction feedback method, equipment and recording medium |
| CN110047520A (en) * | 2019-03-19 | 2019-07-23 | 北京字节跳动网络技术有限公司 | Control method, device, electronic equipment and the computer readable storage medium that audio plays |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106503127B (en) | 2019-09-27 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102473320B (en) | Bringing a visual representation to life via learned input from the user | |
| Collins | Playing with sound: a theory of interacting with sound and music in video games | |
| CN102129343B (en) | Directed performance in motion capture system | |
| US9519989B2 (en) | Visual representation expression based on player expression | |
| Johnson et al. | Sympathetic interfaces: using a plush toy to direct synthetic characters | |
| TWI377055B (en) | Interactive rehabilitation method and system for upper and lower extremities | |
| CN102822869B (en) | Capture view and the motion of the performer performed in the scene for generating | |
| JP2019532374A5 (en) | ||
| Bode | Making believe: Screen performance and special effects in popular cinema | |
| WO2009021124A2 (en) | System and method for a motion sensing amusement device | |
| CN106503127B (en) | Music data processing method and system based on facial action identification | |
| JP7355483B2 (en) | Game systems and programs | |
| JP2018075260A (en) | Game system and program | |
| JP2014023745A (en) | Dance teaching device | |
| JP6839730B2 (en) | Information processing system, information processing method and computer program | |
| Taheri et al. | Exploratory design of a hands-free video game controller for a quadriplegic individual | |
| CN114712862A (en) | Virtual pet interaction method, electronic device and computer-readable storage medium | |
| EP3964271A1 (en) | User input method and apparatus | |
| JP2003088681A (en) | Game information, information storage medium, and game apparatus | |
| JP2020130592A (en) | Program, method, information processing device, and system | |
| Barbulescu et al. | Making movies from make-believe games | |
| TWI824883B (en) | A virtual reality interactive system that uses virtual reality to simulate expressions and emotions for training | |
| Larsson | Discerning emotion through movement: A study of body language in portraying emotion in animation | |
| Junius | Puppitor: Building an Acting Interface for Videogames | |
| US20240042323A1 (en) | Information processing system, method for processing information, and non-transitory computer-readable information storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |