CN108399653A - augmented reality method, terminal device and computer readable storage medium - Google Patents
augmented reality method, terminal device and computer readable storage medium Download PDFInfo
- Publication number
- CN108399653A CN108399653A CN201810068231.4A CN201810068231A CN108399653A CN 108399653 A CN108399653 A CN 108399653A CN 201810068231 A CN201810068231 A CN 201810068231A CN 108399653 A CN108399653 A CN 108399653A
- Authority
- CN
- China
- Prior art keywords
- augmented reality
- face
- image data
- real
- model
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000003190 augmentative effect Effects 0.000 title claims abstract description 124
- 238000000034 method Methods 0.000 title claims abstract description 59
- 238000003860 storage Methods 0.000 title claims abstract description 16
- 239000000284 extract Substances 0.000 claims abstract description 15
- 238000012549 training Methods 0.000 claims description 36
- 238000000605 extraction Methods 0.000 claims description 35
- 238000013527 convolutional neural network Methods 0.000 claims description 15
- 230000006870 function Effects 0.000 claims description 11
- 238000004422 calculation algorithm Methods 0.000 claims description 8
- 238000004891 communication Methods 0.000 claims description 4
- 238000004590 computer program Methods 0.000 claims description 4
- 238000001514 detection method Methods 0.000 claims description 4
- 230000008569 process Effects 0.000 abstract description 14
- 230000002452 interceptive effect Effects 0.000 abstract description 5
- 238000005516 engineering process Methods 0.000 description 8
- 210000000887 face Anatomy 0.000 description 8
- 230000000694 effects Effects 0.000 description 4
- 230000002708 enhancing effect Effects 0.000 description 4
- 238000003062 neural network model Methods 0.000 description 4
- 238000013528 artificial neural network Methods 0.000 description 3
- 230000001815 facial effect Effects 0.000 description 3
- 241000406668 Loxodonta cyclotis Species 0.000 description 2
- 230000006399 behavior Effects 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 238000010801 machine learning Methods 0.000 description 2
- 230000001953 sensory effect Effects 0.000 description 2
- 235000007926 Craterellus fallax Nutrition 0.000 description 1
- 241001269238 Data Species 0.000 description 1
- 240000007175 Datura inoxia Species 0.000 description 1
- 241001465754 Metazoa Species 0.000 description 1
- 230000006978 adaptation Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 235000013399 edible fruits Nutrition 0.000 description 1
- 210000004709 eyebrow Anatomy 0.000 description 1
- 230000000977 initiatory effect Effects 0.000 description 1
- 230000014759 maintenance of location Effects 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 238000009331 sowing Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
- G06T19/006—Mixed reality
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/22—Matching criteria, e.g. proximity measures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T15/00—3D [Three Dimensional] image rendering
- G06T15/10—Geometric effects
- G06T15/20—Perspective computation
- G06T15/205—Image-based rendering
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
- G06V10/443—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components by matching or filtering
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/161—Detection; Localisation; Normalisation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/168—Feature extraction; Face representation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2200/00—Indexing scheme for image data processing or generation, in general
- G06T2200/04—Indexing scheme for image data processing or generation, in general involving 3D image data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30196—Human being; Person
- G06T2207/30201—Face
Abstract
The present invention relates to technical field of computer vision, a kind of augmented reality method, terminal device and computer readable storage medium are disclosed.In the present invention, augmented reality method includes：During playing network streaming media, the augmented reality watching request that user sends is received；According to augmented reality watching request, the rending model for the augmented reality for needing to be superimposed is obtained, and determine the object for needing to carry out augmented reality；The extract real-time image data from network flow-medium determines real-time dynamic information of the object in image data；According to real-time dynamic information of the object in image data, it is superimposed the rending model of augmented reality in real time on the image data, obtains the picture of augmented reality.The augmented reality method provided in embodiment of the present invention, can be such that user participates in Video Applications, keep viewing process more interactive and interesting.
Description
Technical field
The present invention relates to technical field of computer vision, more particularly to a kind of augmented reality method, terminal device and calculating
Machine readable storage medium storing program for executing.
Background technology
With the development of computer communication technology, Internet technology and multimedia technology, the configuration of smart television and property
It can become better and better, and in order to cater to user demand, be also equipped with function of surfing the Net, user is facilitated to obtain the video resource on network
It is watched.In order to make user that can have better visual effect in watching video resource, such as during watching video
Certain special efficacys are added, and then promote user's viewing experience, have the intelligence of AR (augmented reality, Augmented Reality) function
Energy TV comes into being.
AR technologies are former as a kind of by " seamless " the integrated new technology of real world information and virtual world information
This in real world certain time, in spatial dimension be difficult the entity information experienced, such as vision, sound, taste, tactile
Information is perceived virtual Information application to real world by human sensory by science and technology such as computers.Specifically
True environment and virtual object are added to the same picture or space in real time, both make to exist simultaneously, to reach
The sensory experience of exceeding reality, and then promote user's viewing experience.
But inventor has found that at least there are the following problems in the prior art：The existing smart television for having AR functions
Video-see mode does not have interactive function substantially, and user is during watching video, reception video playing that can only be passive
Information can not be participated in adequately in Video Applications so that the individual demand of user is difficult to be met.
Invention content
The purpose of the present invention is to provide a kind of augmented reality method, terminal device and computer readable storage medium, energy
Enough user is made to participate in Video Applications, keeps viewing process more interactive and interesting.
In order to solve the above technical problems, embodiments of the present invention provide a kind of augmented reality method, network is being played
During Streaming Media, the augmented reality watching request that user sends is received；According to augmented reality watching request, obtains and need to fold
The rending model of the augmented reality added, and determine the object for needing to carry out augmented reality；The extract real-time figure from network flow-medium
As data, real-time dynamic information of the object in image data is determined；According to real-time dynamic information of the object in image data,
It is superimposed the rending model of augmented reality in real time on the image data, obtains the picture of augmented reality.
Embodiments of the present invention additionally provide a kind of terminal device, at least one processor；And at least one place
Manage the memory of device communication connection；Wherein, memory is stored with the instruction that can be executed by least one processor, instructs by least
One processor executes, so that at least one processor is able to carry out the augmented reality side involved in arbitrary embodiment of the invention
Method.
Embodiments of the present invention additionally provide a kind of computer readable storage medium, are stored with computer program, and count
Calculation machine program can realize the augmented reality method involved in arbitrary embodiment of the invention when being executed by processor.
Embodiment of the present invention in terms of existing technologies, during playing network streaming media, according to receiving
Augmented reality watching request, determine the object for needing to carry out augmented reality, determining the object real from network flow-medium
When the image data extracted in real-time dynamic information after, it is folded in real time on the image data according to determining real-time dynamic information
The rending model for adding augmented reality obtains the picture of augmented reality.By means of which so that the network that user plays in viewing
When Streaming Media, augmented reality operation can be carried out at any time, and can be determined to need to carry out augmented reality according to personal like
The rending model for the augmented reality that object and needs are superimposed, so that viewing process is more interactive and interesting.
In addition, it is necessary to which the object for carrying out augmented reality is behaved；The extract real-time image data from network flow-medium, determining pair
As the real-time dynamic information in image data, specifically include：As unit of frame from network flow-medium extract real-time picture number
According to；Face datection is carried out to each frame image data, determines real-time dynamic information of the face of object in image data.It is needing
When carrying out the object of augmented reality and behaving, by carrying out Face datection to each frame image data, so as to accurately really
Determine real-time dynamic information of the face of object in image data so that the rending model of augmented reality can be accurately superimposed upon
It needs to carry out on the object of augmented reality, ensures the effect of augmented reality.
In addition, carrying out Face datection to each frame image data, determine that the face of object is real-time dynamic in image data
State information, specifically includes：According to the Face datection model to prestore, Face datection is carried out to each frame image data, obtains image
Real-time dynamic information of all faces in image data in data；Wherein, Face datection model is：Based on convolutional neural networks
Algorithm carries out convolutional neural networks training to face sample data and obtains；According to the face characteristic extraction model to prestore, to image
All faces in data carry out face characteristic extraction；Wherein, face characteristic extraction model is according to the people in face sample data
The training of face feature obtains；The face characteristic extracted is matched with the face characteristic of preset object, determines the people of object
Real-time dynamic information of the face of face and object in image data.
In addition, before playing network streaming media, augmented reality method further includes：Determine face characteristic extraction model；Really
Determine face characteristic extraction model, specifically includes：Training pattern is built according to the face characteristic in face sample data；It will training mould
The convolution kernel that size is 5 × 5 in type is split as the convolution kernel that two sizes are 3 × 3；Based on convolutional neural networks algorithm, to instruction
Practice model to be trained, obtains face characteristic extraction model.During determining face characteristic extraction model, by that will train
The convolution kernel that size is 5 × 5 in model is split as the convolution kernel that two sizes are 3 × 3, the training pattern being then based on after splitting
It is trained, increases the network depth of training pattern so that the extraction accuracy of the face characteristic extraction model trained can
It greatly improves.
In addition, before according to the face characteristic structure training pattern in face sample data, augmented reality method further includes：
Face sample data is normalized.By the way that face sample data is normalized, follow-up training is accelerated
The convergence rate of face characteristic extraction model in the process, and the extensive of face characteristic extraction model is improved to a certain extent
Ability.
In addition, the face characteristic extracted is matched with the face characteristic of preset object, the face of object is determined,
It specifically includes：The face characteristic extracted is matched one by one with the face characteristic in face sample data, and is based on cosine
Function obtains cosine similarity；Cosine similarity is compared with preset similarity threshold, if cosine similarity is more than phase
Like degree threshold value, determine that the corresponding face of face characteristic is the face of object.Face matching is carried out by using cosine similarity, really
The face for determining object greatly reduces the complexity calculated in matching process, to improve matching speed.
In addition, Face datection model is real-time target detection model.In which, Face datection model is examined for real-time target
Model is surveyed, the detection process to image data can be better met, and does not influence user and watches video.
In addition, being superimposed the rending model of augmented reality in real time on the image data, after obtaining the picture of augmented reality, increase
Practical method further includes by force：The identity information for obtaining and showing object carries to facilitate the identity information that user knows object
User experience is risen.
Description of the drawings
One or more embodiments are illustrated by the picture in corresponding attached drawing, these are exemplary
Illustrate not constitute the restriction to embodiment, the element with same reference numbers label is expressed as similar member in attached drawing
Part, unless there are special statement, composition does not limit the figure in attached drawing.
Fig. 1 is the flow chart of the augmented reality method of first embodiment of the invention；
Fig. 2 is the flow chart of the augmented reality method of second embodiment of the invention；
Fig. 3 is the flow chart of the augmented reality method of third embodiment of the invention；
Fig. 4 is the block diagram of the terminal device of four embodiment of the invention.
Specific implementation mode
To make the object, technical solutions and advantages of the present invention clearer, below in conjunction with attached drawing to each reality of the present invention
The mode of applying is explained in detail.However, it will be understood by those skilled in the art that in each embodiment of the present invention,
Many technical details are proposed in order to make reader more fully understand the present invention.But even if without these technical details and base
In the various changes and modifications of following embodiment, each claim technical side claimed of the present invention can also be realized
Case.
The first embodiment of the present invention is related to a kind of augmented reality methods.In the present embodiment, playing network streaming matchmaker
Body can be the terminal devices such as intelligent TV set, mobile phone, tablet computer, and the network flow-medium of broadcasting can be various video texts
Part, such as a film, a TV play, the specific implementation flow of augmented reality method are as shown in Figure 1.
In a step 101, the augmented reality watching request that user sends is received.
In a step 102, the rending model for the augmented reality for needing to be superimposed is obtained.
In step 103, the object for needing to carry out augmented reality is determined.
At step 104, the extract real-time image data from network flow-medium determines that object is real-time in image data
Multidate information.
In step 105, it is superimposed the rending model of augmented reality in real time on the image data, obtains the picture of augmented reality
Face.
By above-mentioned flow it is not difficult to find that augmented reality method in present embodiment, specifically in playing network streaming matchmaker
During body, the augmented reality watching request that user sends is received, obtains the rending model for needing the augmented reality being superimposed, root
According to augmented reality watching request, the determining object for needing to carry out augmented reality, the extract real-time image data from network flow-medium,
Real-time dynamic information of the object in image data is determined, according to real-time dynamic information of the object in image data, in image
The rending model of augmented reality is superimposed in data in real time, obtains the picture of augmented reality.
Wherein, real-time dynamic information includes the basic informations such as position and size, in the other embodiment of the present invention, is
Object in the more intelligent watching request to augmented reality carries out the superposition of rending model, also includes pair in real-time dynamic information
As this information of state, to be more accurately overlapped setting to rending model, by taking to ask object be face as an example, work as determination
The position of face and size and then the real-time status according to face, such as positive face or side face carry out the folded of rending model
Add.
It should be noted that in the present embodiment, it is two dimension or three-dimensional to need the rending model for the augmented reality being superimposed
Dynamic virtual model, such as a piece of ocean, a butterfly danced in the air.
In addition, user uses for convenience, need the rending model for the augmented reality being superimposed that can be managed by user,
For example user can like downloading satisfactory rending model from network in advance according to individual, and can be according to personal like
It deletes, the rending model etc. that update is local so that user can freely customize the special efficacy of augmented reality to be achieved.
In order to make it easy to understand, being illustrated below：
Such as during playing video using intelligent TV set, if user needs to currently playing video
Augmented reality is carried out, then it is existing can to recall enhancing at the interface of intelligent TV set by the remote control equipment (such as remote controler) of offer
Real function menu, while the wash with watercolours for the augmented reality that selection will carry out the object of augmented reality in the menu and needs are superimposed
Contaminate model.
In addition, it is necessary to which explanation, in the present embodiment, selection need to carry out the specific behaviour of the object of augmented reality
Make, can be specifically by the directionkeys on remote controler, the arrow on moving boundary choose in current picture some or it is several
A object, such as house, animal etc..
In doing so, the letter carried in the augmented reality watching request that smart television chance is sent according to user
Breath obtains the rending model of user's selection from local or network, and determines the object for needing to carry out augmented reality.
After determining the object for needing to carry out augmented reality, the extract real-time image data from network flow-medium, determining should
Real-time dynamic information of the object in image data, so as to be believed according to real-time dynamic of the determining object in image data
Breath, in playing process, the rending model for the augmented reality that superposition obtains in real time on the image data is wanted to obtain user
Augmented reality effect.
It is limited it should be noted that these are only for example, not constituted to protection scope of the present invention.Actually answering
In, user can initiate enhancing watching request at any time during watching video, enhance the initiation of watching request, render
The selection of model and augmented reality object can use crowd, occasion by those skilled in the art according to terminal device
It is configured, is not limited this time.
Compared with prior art, the augmented reality method provided in present embodiment so that the net that user plays in viewing
When network Streaming Media, augmented reality operation can be carried out at any time, and can be determined to need to carry out augmented reality according to personal like
Object and the rending model of augmented reality that is superimposed of needs so that viewing process is more interactive and interesting.
Second embodiment of the present invention is related to a kind of augmented reality method.Base of the present embodiment in first embodiment
It is further improved on plinth, main improvements are：Need carry out augmented reality object behave when, as unit of frame from
Extract real-time image data in network flow-medium, and Face datection is carried out to each frame image data, determine that the face of object exists
Real-time dynamic information in image data, wherein real-time dynamic information include the basic informations such as position and size, the present invention's
In other embodiment, in order to which the object in the more intelligent watching request to augmented reality carries out the superposition of rending model, in real time
Also include this information of Obj State in multidate information.The detailed process of the augmented reality method provided in present embodiment is as schemed
Shown in 2.
Specifically, in the present embodiment, including step 201 is to step 208, wherein step 201 to step 203 is divided
It is not roughly the same to step 103 with the step 101 in first embodiment, step 208 and the step 105 in first embodiment
Roughly the same, details are not described herein again, mainly introduces difference below：
In step 204, as unit of frame from network flow-medium extract real-time image data.
Specifically, in present embodiment, determining that the object for needing to carry out augmented reality is real-time in image data
When position, by as unit of frame from network flow-medium extract real-time image data so that determine real-time dynamic information more
Accurately.
In step 205, according to the Face datection model to prestore, all faces are obtained in image data in image data
Real-time dynamic information.
Specifically, in the present embodiment, the behaviour of real-time dynamic information of the face of object in image data is determined
Make, need according to the Face datection model to prestore, Face datection is carried out to each frame image data.
In addition, from each frame image data extracted in network flow-medium, often there is more than one object, such as one
There can be multiple performers in a certain picture of portion's film.Object implementatio8 in order to be accurately user's selection shows the effect enhanced
Fruit determines that real-time dynamic of the face of the object of user's selection in image data is believed according to the Face datection model to prestore
It when breath, needs first to obtain real-time dynamic information of all faces in image data in current image date, is then directed to all
Face carry out face characteristic extraction, that is, enter step 206 operation.
It should be noted that the Face datection model in present embodiment is：Based on convolutional neural networks algorithm to face
Sample data carries out convolutional neural networks training and obtains.
About convolutional neural networks, those skilled in the art could be aware that, in machine learning, before being a kind of depth
Artificial neural network is presented, can accurately identify the information in image.Therefore, by (such as advance to face sample data
Downloaded from network, or shoot the facial image of typing) convolutional neural networks training is carried out, can accurately it know to obtain one
Do not go out the Face datection model in image.
In addition, it is noted that the Face datection model used in the present embodiment is specifically using Caffe (volumes
Product neural network framework, Convolutional Architecture for Fast Feature Embedding) it builds, so
The training of convolutional neural networks algorithm is based on afterwards to obtain.Since Caffe is a clear, readable high, quick deep learning frame
Therefore frame obtains Face datection model to train based on Caffe, can greatly improve the speed of service, and can greatly reduce
The size of the Face datection model of training gained so that the Face datection model of training gained is that real-time target detects mould.
In addition, in order to promote Face datection speed, the Face datection model in present embodiment is specifically to use existing people
Face data carry out convolution god compared with the face recognition database of horn of plenty to real-time target detection (YOLO) neural network model
It is obtained through network training.
Further, in order to ensure that the Face datection model of training acquisition can be more accurate, the liter of YOLO may be used
Grade version YOLOv2 neural network models carry out convolutional neural networks and train to obtain.
For the ease of understanding the training of Face datection model, it is illustrated below：
It writes script and face sample data set is changed into the recognizable formats of YOLO, configure three master files of YOLO
Myobj.data (for storing the face sample data for being converted to YOLO and can recognize that format), myobj.name are (every for storing
The corresponding title of a face sample data), myobj.cfg (for storing relevant parameter needed for training process), in YOLO
Official website, which downloads weights file and then runs to order, to be started to train, and observes the change of avg (average value of each training result) this value
Change, if this value no longer becomes smaller substantially, training can stopped, and Face datection can be carried out by just obtaining one at this time
Face datection model.
It is limited it should be noted that these are only for example, not constituted to protection scope of the present invention.Actually answering
In, those skilled in the art can obtain the present invention in fact according to its well known computer vision, image processing method training
The neural network model needed for mode is applied, is not limited herein.
In step 206, according to the face characteristic extraction model to prestore, face is carried out to all faces in image data
Feature extraction.
Specifically, when carrying out face characteristic extraction, specifically according to the face characteristic extraction model to prestore, to image
All faces in data carry out face characteristic extraction.
In addition, the face characteristic extraction model in present embodiment is obtained according to the face characteristic training in face sample data
.
In the present embodiment, in order to increase the network depth of training pattern so that the face characteristic extraction trained
The extraction accuracy of model can greatly improve, and specifically training obtains face characteristic extraction model in the following ways, specific as follows：
Training pattern is built according to the face characteristic in face sample data, the convolution for being 5 × 5 by size in training pattern
Core is split as the convolution kernel that two sizes are 3 × 3, is based on convolutional neural networks algorithm, is trained to training pattern, obtains people
Face Feature Selection Model.
It should be noted that the face characteristic extraction model in present embodiment and a kind of convolutional neural networks model,
The face Feature Selection Model is mainly made of convolutional layer, pond layer and full articulamentum, wherein the combination of convolutional layer and pond layer
Can occur repeatedly, after full articulamentum is located at pond layer, the output layer as entire model.
In addition, face characteristic described in present embodiment, specifically uses in face characteristic extraction model and is saved in output layer
The output of point is as face characteristic, which can be made of each characteristic point of face, such as eyes, nose, mouth
The profile point of angle point, eyebrow and face's other component.
In addition, it is necessary to explanation, in practical applications, full articulamentum can there are two, if full articulamentum is two,
Then output layer is the second full articulamentum, and specific those skilled in the art can be arranged as required to, not be limited herein.
In addition, it is noted that in order to accelerate in follow-up training process, the convergence rate of face characteristic extraction model,
And generalization ability (adaptation energy of the machine learning algorithm to fresh sample of face characteristic extraction model is promoted to a certain extent
Power), the face characteristic extraction model used in present embodiment is carried out being trained according to the face characteristic in face sample data
Before training, normalized is carried out to face sample data, to greatly reduce in training process in every layer of convolutional layer
Convolution kernel and as the number of nodes in the full articulamentum of output layer, simplifies the various calculating in training process.
In step 207, face characteristic matching operation is carried out to the face characteristic that extracts, determines the face of object and right
Real-time dynamic information of the face of elephant in image data.
Specifically, in present embodiment, by the way that the face in the face characteristic extracted and face sample data is special
Sign is matched one by one, and is based on cosine function, obtains cosine similarity.
Further, by the way that cosine similarity to be compared with preset similarity threshold, if cosine similarity is more than
Similarity threshold, it is determined that the corresponding face of face characteristic is the face of object.
In addition, it is necessary to explanation, in the real-time dynamic information according to the face of object in image data, in picture number
According to the rending model of upper real-time superposition augmented reality, obtain in the picture of augmented reality, present embodiment is specifically to use
OpenGL (OpenGL, Open Graphics Library) 3-D graphic API (application programming interface,
Application Programming Interface) subset OpenGL ES (OpenGL for Embedded
Systems) the rending model for the augmented reality being superimposed to needs, such as two dimension or Three-Dimensional Dynamic dummy model are rendered, according to
Face key point determines facial orientation so as to adjust the three-dimensional position of rending model.
Adjust rending model three-dimensional position operation, can be specifically：After obtaining the azimuth of facial orientation, with this
Azimuth rotates the position of rending model around Y-axis, by rotating to obtain three-dimension object on two dimensional surface to projection matrix
Coordinate position, then paster is rendered, to obtain that there is the picture of relief augmented reality.
In addition, in order to further facilitate user, user experience is promoted, the Face datection model used in present embodiment,
Face characteristic extraction model and face sample data etc. can be downloaded and deleted according to personal like by user, from
And the augmented reality effect for making user that can freely customize the video file of augmented reality to be carried out and reach.
By foregoing description it is not difficult to find that compared with prior art, the augmented reality method provided in present embodiment is led to
The extract real-time image data from network flow-medium is crossed as unit of frame, and according to the Face datection model to prestore, to each frame
Image data carries out Face datection, real-time dynamic information of all faces in image data in image data is obtained, according to pre-
The face characteristic extraction model deposited carries out face characteristic extraction, finally the people to extracting to all faces in image data
Face feature carries out face characteristic matching operation, the real-time dynamic letter of the face of the face and object that determine object in image data
Breath, to user select need progress augmented reality object for some particular person when, can accurately determine that this is right
The real-time dynamic information of the face of elephant and the face of object in image data, further promotes user experience.
In addition, it is necessary to explanation, due to the augmented reality method that present embodiment provides, primarily directed to smart television
Video that machine sowing is put, such as TV play, film etc. carry out augmented reality operation, and TV play, film are made in advance, inner
The role in face knows in advance, thus can to the human face data of the performer arrived involved in the videos such as TV play, film into
The training of row neural network realizes that the enhancing to personage in video is existing to obtain to identify the Face datection model of face
Practical operation is made.
In addition, since the featured performer of each TV play or film is fixed and small, targetedly
Be trained, the neural network model for obtaining each TV play or film is smaller, and computational complexity is relatively low, therefore will not
Influence the fluency of broadcasting video.
Third embodiment of the present invention is related to a kind of augmented reality method.Base of the present embodiment in second embodiment
It is further improved on plinth, specific improvements are：It is superimposed the rending model of augmented reality in real time on the image data, obtains
After the picture of augmented reality, it can also further obtain and show that the identity information of object, detailed process are as shown in Figure 3.
Specifically, in the present embodiment, including step 301 is to step 309, wherein step 301 to step 308 is divided
Not roughly the same to step 208 with the step 201 in second embodiment, details are not described herein again, mainly introduces difference below
Place：
In a step 309, obtain and show the identity information of object.
Specifically, network flow-medium described in present embodiment can be specifically a film or a TV play,
Accordingly, it is determined that need carry out augmented reality object would generally be one of performer.And the information of most of performer is
It can be arrived by web search, thus obtain object identity information, i.e., the mode of the identity information of some performer specifically can be with
It is obtained online by network, for example the body of the performer is obtained by internet hunt according to the face characteristic of the performer extracted
Part information, such as name, age and the works associated summary performed.
In addition, whether film or TV play, performer's quantity therein is all limited.Therefore, it is somebody's turn to do in user's viewing
When film, the identity information of related performer in the film can be cached to terminal device local, need to drill a certain in user
Member obtains corresponding body when carrying out augmented reality, and showing its identity information, according to related face characteristic from local
Part information is shown.
It is limited it should be noted that these are only for example, not constituted to protection scope of the present invention.Actually answering
In, above-mentioned function can need to select according to user, for example be supplied to the manipulable interface of user, be decided whether by user
It needs after carrying out augmented reality, shows that the identity information of augmented reality object, concrete implementation mode are not limited herein.
Compared with prior art, the augmented reality method provided in present embodiment, the picture for obtaining augmented reality it
Afterwards, by the identity information obtained and display carries out the object of enhancing display on interface, the body that user knows object is facilitated
Part information, the user experience is improved.
The 4th embodiment of the present invention is related to a kind of terminal device, and concrete structure is as shown in Figure 4.
The terminal device includes one or more processors 401 and memory 402, is with a processor 401 in Fig. 4
Example.
In the present embodiment, processor 401 can be connected with memory 402 by bus or other modes, in Fig. 4 with
For being connected by bus.
Memory 402 is used as a kind of computer readable storage medium, can be used for storing software program, computer can perform journey
Sequence and module, the corresponding program instruction/module of augmented reality method as involved in any means embodiment of the present invention.Place
Reason device 401 is stored in software program, instruction and module in memory 402 by operation, so that execute server is various
The augmented reality method involved in any means embodiment of the present invention is realized in application of function and data processing.
Memory 402 may include storing program area and storage data field, wherein storing program area can store operation system
System, the required application program of at least one function；Storage data field can store various face sample datas and trained
Various models, such as convolutional neural networks model, face characteristic privilege mode.In addition, memory 402 may include that high speed is random
Memory is accessed, can also include memory, for example, at least a disk memory, flush memory device or other solid-state storages
Device.In some embodiments, it includes the memory remotely located relative to processor 401 that memory 402 is optional, these are remote
Journey memory can pass through network connection to terminal device.The example of above-mentioned network includes but not limited to internet, enterprises
Net, LAN, mobile radio communication and combinations thereof.
In practical applications, the instruction of the execution of at least one processor 401 can be stored in memory 402, instructed by extremely
A few processor 401 executes, so that at least one processor 401 is able to carry out what any means embodiment of the present invention was related to
Augmented reality method realizes augmented reality, not the technical detail of detailed description in the present embodiment, reference can be made to the present invention is implemented
The augmented reality method that mode is provided.
The 5th embodiment of the present invention is related to a kind of computer readable storage medium, in the computer readable storage medium
It is stored with computer instruction, which enables a computer to execute the increasing involved in any means embodiment of the present invention
Strong practical method.One of ordinary skill in the art will appreciate that realize all or part of flow in above method embodiment,
It is that relevant hardware can be instructed to complete by computer program, is stored in computer readable storage medium in practical applications
In computer program may include the flow of above-mentioned any means embodiment.Wherein, storage medium can be magnetic disc, CD, only
Read storage memory (Read-Only Memory, ROM) or random access memory (Random AccessMemory, RAM)
Deng.
It will be understood by those skilled in the art that the respective embodiments described above are to realize the specific embodiment party of the present invention
Formula, and in practical applications, can to it, various changes can be made in the form and details, without departing from the spirit and model of the present invention
It encloses.
Claims (10)
1. a kind of augmented reality method, which is characterized in that including：
During playing network streaming media, the augmented reality watching request that user sends is received；
According to the augmented reality watching request, the rending model for the augmented reality for needing to be superimposed is obtained, and determination needs to carry out
The object of augmented reality；
The extract real-time image data from the network flow-medium determines real-time dynamic of the object in described image data
Information；
According to real-time dynamic information of the object in described image data, it is superimposed the increasing in real time in described image data
The rending model of strong reality, obtains the picture of augmented reality.
2. augmented reality method according to claim 1, which is characterized in that it is described need carry out augmented reality object be
People；
The extract real-time image data from the network flow-medium determines real-time dynamic of the object in described image data
Information specifically includes：
As unit of frame from the network flow-medium extract real-time image data；
Face datection is carried out to each frame described image data, determines that the face of the object is real-time in described image data
Multidate information.
3. augmented reality method according to claim 2, which is characterized in that described to be carried out to each frame described image data
Face datection determines real-time dynamic information of the face of the object in described image data, specifically includes：
According to the Face datection model to prestore, Face datection is carried out to each frame described image data, obtains described image data
In real-time dynamic information of all faces in described image data；Wherein, the Face datection model is：Based on convolutional Neural
Network algorithm carries out convolutional neural networks training to face sample data and obtains；
According to the face characteristic extraction model to prestore, face characteristic extraction is carried out to all faces in described image data；Its
In, the face characteristic extraction model is trained according to the face characteristic in the face sample data and is obtained；
The face characteristic extracted is matched with the face characteristic of the preset object, determines the people of the object
Real-time dynamic information of the face of face and the object in described image data.
4. augmented reality method according to claim 3, which is characterized in that before playing network streaming media, the increasing
Practical method further includes by force：
Determine the face characteristic extraction model；
The determination face characteristic extraction model, specifically includes：
Training pattern is built according to the face characteristic in the face sample data；
The convolution kernel that size in the training pattern is 5 × 5 is split as the convolution kernel that two sizes are 3 × 3；
Based on the convolutional neural networks algorithm, the training pattern is trained, obtains the face characteristic extraction model.
5. augmented reality method according to claim 4, which is characterized in that described according in the face sample data
Before face characteristic builds training pattern, the augmented reality method further includes：
The face sample data is normalized.
6. the augmented reality method according to claim 3 to 5 any one, which is characterized in that the institute that will be extracted
It states face characteristic to be matched with the face characteristic of the preset object, determines the face of the object, specifically include：
The face characteristic extracted is matched one by one with the face characteristic in the face sample data, and based on remaining
String function obtains cosine similarity；
The cosine similarity is compared with preset similarity threshold, if the cosine similarity is more than the similarity
Threshold value determines that the corresponding face of the face characteristic is the face of the object.
7. the augmented reality method according to claim 3 to 5 any one, which is characterized in that the Face datection model
For real-time target detection model.
8. the augmented reality method according to claim 1 to 5 any one, which is characterized in that in described image data
It is superimposed the rending model of the augmented reality in real time, after obtaining the picture of augmented reality, the augmented reality method further includes：
Obtain and show the identity information of the object.
9. a kind of terminal device, which is characterized in that including：
At least one processor；And
The memory being connect at least one processor communication；Wherein,
The memory is stored with the instruction that can be executed by least one processor, and described instruction is by least one place
It manages device to execute, so that at least one processor is able to carry out the augmented reality side as described in claim 1 to 8 any one
Method.
10. a kind of computer readable storage medium, which is characterized in that realized when the computer program is executed by processor as weighed
Profit requires the augmented reality method described in 1 to 8 any one.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810068231.4A CN108399653A (en) | 2018-01-24 | 2018-01-24 | augmented reality method, terminal device and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810068231.4A CN108399653A (en) | 2018-01-24 | 2018-01-24 | augmented reality method, terminal device and computer readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN108399653A true CN108399653A (en) | 2018-08-14 |
Family
ID=63094205
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810068231.4A Pending CN108399653A (en) | 2018-01-24 | 2018-01-24 | augmented reality method, terminal device and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108399653A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109785094A (en) * | 2018-12-14 | 2019-05-21 | 深圳壹账通智能科技有限公司 | Methods of exhibiting, terminal device and the computer readable storage medium of revenue and expenditure information |
| CN111246118A (en) * | 2020-04-27 | 2020-06-05 | 成都派沃特科技股份有限公司 | Display method, device and equipment of AR element and storage medium |
| CN111833461A (en) * | 2020-07-10 | 2020-10-27 | 北京字节跳动网络技术有限公司 | Method and device for realizing special effect of image, electronic equipment and storage medium |
| CN113127126A (en) * | 2021-04-30 | 2021-07-16 | 上海哔哩哔哩科技有限公司 | Object display method and device |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105117692A (en) * | 2015-08-05 | 2015-12-02 | 福州瑞芯微电子股份有限公司 | Real-time face identification method and system based on deep learning |
| CN105872442A (en) * | 2016-03-30 | 2016-08-17 | 宁波三博电子科技有限公司 | Instant bullet screen gift giving method and instant bullet screen gift giving system based on face recognition |
| CN107437272A (en) * | 2017-08-31 | 2017-12-05 | 深圳锐取信息技术股份有限公司 | Interaction entertainment method, apparatus and terminal device based on augmented reality |
-
2018
- 2018-01-24 CN CN201810068231.4A patent/CN108399653A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105117692A (en) * | 2015-08-05 | 2015-12-02 | 福州瑞芯微电子股份有限公司 | Real-time face identification method and system based on deep learning |
| CN105872442A (en) * | 2016-03-30 | 2016-08-17 | 宁波三博电子科技有限公司 | Instant bullet screen gift giving method and instant bullet screen gift giving system based on face recognition |
| CN107437272A (en) * | 2017-08-31 | 2017-12-05 | 深圳锐取信息技术股份有限公司 | Interaction entertainment method, apparatus and terminal device based on augmented reality |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109785094A (en) * | 2018-12-14 | 2019-05-21 | 深圳壹账通智能科技有限公司 | Methods of exhibiting, terminal device and the computer readable storage medium of revenue and expenditure information |
| CN111246118A (en) * | 2020-04-27 | 2020-06-05 | 成都派沃特科技股份有限公司 | Display method, device and equipment of AR element and storage medium |
| CN111246118B (en) * | 2020-04-27 | 2020-08-21 | 成都派沃特科技股份有限公司 | Display method, device and equipment of AR element and storage medium |
| CN111833461A (en) * | 2020-07-10 | 2020-10-27 | 北京字节跳动网络技术有限公司 | Method and device for realizing special effect of image, electronic equipment and storage medium |
| CN111833461B (en) * | 2020-07-10 | 2022-07-01 | 北京字节跳动网络技术有限公司 | Method and device for realizing special effect of image, electronic equipment and storage medium |
| CN113127126A (en) * | 2021-04-30 | 2021-07-16 | 上海哔哩哔哩科技有限公司 | Object display method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20200404219A1 (en) | Immersive interactive remote participation in live entertainment | |
| US10719192B1 (en) | Client-generated content within a media universe | |
| CN103902489B (en) | Generate and execute the method and system of the Miniapp of computer application | |
| US11514690B2 (en) | Scanning of 3D objects with a second screen device for insertion into a virtual environment | |
| CN108399653A (en) | augmented reality method, terminal device and computer readable storage medium | |
| CN106713988A (en) | Beautifying method and system for virtual scene live | |
| CN106659937A (en) | User-generated dynamic virtual worlds | |
| CN107638690A (en) | Method, device, server and medium for realizing augmented reality | |
| CN109035415B (en) | Virtual model processing method, device, equipment and computer readable storage medium | |
| CN107767438A (en) | A kind of method and apparatus that user mutual is carried out based on virtual objects | |
| CN109743584B (en) | Panoramic video synthesis method, server, terminal device and storage medium | |
| US11285390B2 (en) | Artificial intelligence (AI) controlled camera perspective generator and AI broadcaster | |
| CN109314802A (en) | Game based on position in game is carried out with application | |
| CN113709543A (en) | Video processing method and device based on virtual reality, electronic equipment and medium | |
| JP6379107B2 (en) | Information processing apparatus, control method therefor, and program | |
| CN113392690A (en) | Video semantic annotation method, device, equipment and storage medium | |
| US20230154115A1 (en) | Method and apparatus for providing multi-user-involved augmented reality content for diorama application | |
| CN108421240A (en) | Court barrage system based on AR | |
| US11494964B2 (en) | 2D/3D tracking and camera/animation plug-ins | |
| CN107871338B (en) | Real-time, interactive rendering method based on scene decoration | |
| CN110719415A (en) | Video image processing method and device, electronic equipment and computer readable medium | |
| US20230059361A1 (en) | Cross-franchise object substitutions for immersive media | |
| CN117769823A (en) | In-game asset tracking using NFT tracking impressions across multiple platforms | |
| US11684852B2 (en) | Create and remaster computer simulation skyboxes | |
| TWM497315U (en) | A kind perception of touch with the augmented reality functionality and device tags |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20180814 |
|
| RJ01 | Rejection of invention patent application after publication |