CN110211240A - A kind of augmented reality method for exempting from sign-on ID - Google Patents
A kind of augmented reality method for exempting from sign-on ID Download PDFInfo
- Publication number
- CN110211240A CN110211240A CN201910467466.5A CN201910467466A CN110211240A CN 110211240 A CN110211240 A CN 110211240A CN 201910467466 A CN201910467466 A CN 201910467466A CN 110211240 A CN110211240 A CN 110211240A
- Authority
- CN
- China
- Prior art keywords
- information
- data
- model
- augmented reality
- client
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/013—Eye tracking input arrangements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0487—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/16—Sound input; Sound output
- G06F3/167—Audio in a user interface, e.g. using voice commands for navigating, audio feedback
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
- G06T19/006—Mixed reality
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Data Mining & Analysis (AREA)
- Bioinformatics & Computational Biology (AREA)
- Health & Medical Sciences (AREA)
- Evolutionary Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Artificial Intelligence (AREA)
- Multimedia (AREA)
- Evolutionary Computation (AREA)
- Audiology, Speech & Language Pathology (AREA)
- General Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Computer Graphics (AREA)
- Computer Hardware Design (AREA)
- Software Systems (AREA)
- Processing Or Creating Images (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The invention belongs to augmented reality fields, disclose a kind of augmented reality method for exempting from sign-on ID, this method is with C/S (client-server end) framework, information transmission is carried out using udp protocol, client provides the dynamic loading function of human-computer interaction, information collection and dummy model, the information received is carried out identification classification by the convolutional neural networks of transfer learning training by server-side, dummy model is provided, to realize the effect of augmented reality.Human-computer interaction includes the present invention using by dummy object model upload service end and by client dynamically load, and while the memory needed for reducing client application, the load and interaction of a variety of models can also be realized in the case where not updating client；Solve the problems, such as that traditional augmented reality is demanding to new scene bad adaptability, and exploitation high to mark dependence.This method is applicable in the augmented reality application of a large amount of dummy models, especially in engineering model field.
Description
Technical field
The invention belongs to augmented reality fields, and in particular to a kind of augmented reality method for exempting from sign-on ID.
Background technique
Augmented reality, it is a kind of by " seamless " the integrated new technology of real world information and virtual world information,
It is entity information (visual information, sound, the taste that script is difficult to experience in the certain time spatial dimension of real world
Road, tactile etc.), it by science and technology such as computers, is superimposed again after analog simulation, by virtual Information application to real world, quilt
Human sensory is perceived, to reach the sensory experience of exceeding reality.True environment and virtual object are added in real time
Same picture or space exist simultaneously.
The specific implementation of augmented reality method and is placed to virtual information by carrying out three-dimensional registration in real world
It on three-dimensional site, is finally shown by display equipment, developing AR by this method, (Augmented Reality, enhancing are existing
It is real) application item, it needs to be placed in identification information and virtual information in advance, causes to identify specific mark in use
Information, and new virtual information can not be loaded, it can not be interacted with new virtual information.In addition, AR (Augmented
Reality, augmented reality) objective factors such as angle, distance and ambient of equipment can all influence the tracking of identification information
The effect of identification and model load.
Summary of the invention
It is needed for above-mentioned tradition AR (Augmented Reality, augmented reality) technology identification information and virtual information
It to be placed in advance, the identification of identification information, tracking can be by the interference of extraneous factor, thus the problem of influencing user experience, herein
Propose a kind of augmented reality method for exempting from sign-on ID.This method is suitable for the virtual information of dynamically load target information, uses
In augmented reality application.
In order to solve the above technical problems, the technical solution adopted by the present invention are as follows:
A kind of augmented reality method for exempting from sign-on ID, the augmented reality method are using C/S (Client-Server)
The framework at client-server end carries out information transmission using udp protocol；Client provides human-computer interaction, information collection and void
The dynamic loading function of analog model, server-side know the information received by the convolutional neural networks of transfer learning training
Do not classify, dummy model is provided, to realize the effect of augmented reality.
By dummy object model upload service end and by client dynamically load in the present invention, client application institute is being reduced
While needing memory, the load and interaction of a variety of models can also be realized in the case where not updating client；
The convolutional neural networks identification nicety of grading of transfer learning training is high, speed is fast, and higher generalization ability makes
Identification object is also no longer limited to specific identifier；
In terms of human-computer interaction, interacted in a manner of by conveniently gesture, voice and staring with model, Yong Hucao
Make comfortable natural.
Further, the human-computer interaction, information collection of the client and the dynamic loading function of dummy model include
Information collection function stares identification function, gesture identification function, speech identifying function, space reflection function and dynamically load function
Energy.
Still further, the information collection function of the client, stare identification function, gesture identification function, voice know
Other function, space reflection function and dynamic loading function are realized using following steps:
The information collection function of client described in step C1.；It is taken pictures using holographic glasses HoloLens to target, it will
Shooting result is saved with the format of Jpg；The picture that will take pictures is converted to Sprite format and is loaded onto provided by Uity3D engine
It is shown on UI carrier；
Client described in step C2. stares identification function；It is described stare identification be based on eye movement tracer technique, for
Track and selected holographic object, according to user's head position and direction by Unity3D engine Physical Raycast object
Ricoh's line obtains the feedback of collision result, the letter of position and colliding object including the point of impingement after colliding with holographic object
Breath, to realize the tracking of holographic object in scene and select；Client application can be realized by staring identification function
The selected and movement of dummy object.
The gesture identification function of client described in step C3.；The gesture identification is to identify and track the position of user hand
With capture input gesture while state, system automatic trigger is fed back accordingly, to manipulate the virtual objects in scene；
The speech identifying function of client described in step C4.；The speech recognition is user with the method for voice and holography
Object interaction, speech recognition are realized and keyword and corresponding feedback behavior is arranged in client application,
When user says keyword, client application responds preset feedback behavior；
The space reflection functional development of client described in step C5.；The space reflection is by virtual world and real generation
Boundary is superimposed, is realized by following methods；
Step C5.1 is by using the holographic glasses HoloLens depth camera being equipped with and environment sensing camera, scanning
Environmental data and built-in triangulation around user obtain real world to realize the modelling and digitlization of real world
Digital physical space information；
Step C5.2 calculates whether digital physical space obtained above can place virtual hologram object in real time；By visitor
The space reflection function at family end, the spatial position of dummy model no longer by identification information in real world position constraint；It adopts
Method with space reflection can get rid of the physical location limitation of mark, keep virtual information more quasi- for identifying tracking
Really, it is reasonably combined with real world.
The dynamic loading function of client models described in step C6.；The dynamically load of the model is using holographic glasses
HoloLens loads the method for dummy model by access server to realize.
Further, the input gesture in the step 3 includes Air-tap, Navigation gesture and Bloom
Three kinds.
Further, in the step C6 server store dummy model be by Unity3D engine, in advance will be empty
The compressed package that analog model and script are packaged as AssetBundle is uploaded to server；Holographic glasses HoloLens is according to service
It is after the identification of end to download the AssetBundle compressed package of corresponding model as a result, accessing server and decompress, thus implementation model
Dynamically load.The type of object is thousands of in daily life, is difficult to shift to an earlier date on all objects in client application
Merging, and holographic glasses are compared with high performance computer, rendering capability, memory and performance are all extremely limited, so wearing
Formula augmented reality glasses can not carry a large amount of model for load, therefore present invention employs the dynamically loads from server
The method of model.
Further, the server-side carries out the information received by the convolutional neural networks of transfer learning training
Identification classification, provides dummy model；The convolutional neural networks of the transfer learning training, are realized especially by following steps:
Step S1. establishes sample data set；Good sample data set is the basis of information Classification and Identification, passes through internet
Channel obtains sample image, and sample image is rotated by 90 ° by quantitative proportion, rotates 180 °, horizontal mirror image and vertical mirror image
Exptended sample data set is operated, after expanding, is finally made as the data set identified for information；
Step S2. carries out model training on the data set for information identification that step S1 finally makes, from inhomogeneity
Each sample image for selecting 70%-80% at random is training dataset in not, and remaining sample image is as test data set, repeatedly
Generation number is 40-100 times, carries out model training；
Step S3. is trained by the accuracy rate that penalty values, over-fitting ratio and test data are classified come scoring model
Effect；Wherein, shown in the accuracy rate such as formula (1) of test data classification
In formula (1), ExactQuantity indicates the correct quantity of test data classification results, and TotalQuantity is indicated
The total quantity of test data；The accuracy rate of test data classification is higher, indicates that the effect of network model classification is better；
Obtained by cross entropy loss function of the penalty values by Softmax, as shown in formula (2)
Wherein,1{yi=j } refer to indicative function, its value is 1 when value is true within " { } ", otherwise for
0；Penalty values indicate that the training result of network model is better closer to 0；
Shown in over-fitting ratio such as formula (3)
In formula, TrainAcc indicates training data accuracy rate, and TrainAcc is as shown in Equation 4
In formula, TrainExactQuantity is the correct quantity of training data classification results,
TrainTotalQuantity is the total quantity of training data.Over-fitting ratio indicates the extensive energy of network model closer to 1
Power is better.
Further, the model training process of the step S2 are as follows:
Step S2.1AlexNet network model carries out pre-training on the data set of ImageNet, will by the step
The parameter initialization of AlexNet network model；
Step S2.2 is since last three layers of AlexNet network model are configured as 1000 classification, so by last three
The full articulamentum of layer carries out re -training to adapt to new classification, and the parameter of new full articulamentum, Lai Shiying are retained by the step
The class categories for the data set that step S1 is established；
Step S2.3 joins first five layer of convolutional layer and its corresponding pond layer, the activation primitive and model in step S2.1
Several full articulamentums and its parameter in step S2.2 combine, and are finely adjusted to complete the training of model.
Further, the information transmission is by the information of transmitting terminal by processing, is carried out using udp protocol to information
Transmission, the information received is handled and is restored by receiving end；The maximum number of byte that can be transmitted according to its single first will pass
Defeated information is pre-processed, transmitting terminal according to file absolute path go obtain picture, then to picture carry out data encoding,
Data are cut and the operation of addition header information, add file type, file data length, number of data packets in header information
And data number, receiving end according to data header information complete data decoding and recombination, and verify whether null data, if having
Packet loss by verification information back and applies for that transmitting terminal numbers retransmission according to header data to packet loss information.
Further, the information processing of the transmitting terminal includes the following steps:
Step F1. encodes the information of transmission, and according to its information type, type content is encoded, and will compile
In the file type of the result insertion header of code；
Step F2. counts the result length after the information coding of transmission, and the resultant content of statistics is compiled
Code, will be in the file size of the result insertion header of coding；
Result equalization after the information coding of transmission is divided into multiple groups by step F3., and by the content of total number packets amount into
Row coding, will be in the number of data packets of the result insertion header of coding；
Data group after segmentation is numbered step F4. in order, and number content is encoded, the result of coding
It is inserted into the data number of header；
Step F5. repeats step F1 to step F4, pre-processes and retransmits in order to the information after segmentation, it is ensured that information is not
It a large amount of simultaneously can send.
Further, the information processing of the receiving end includes the following steps:
Step R1. decodes the datagram header data received, by file type, IP and source port number classify, together
When create new thread and go to receive new information；
Step R2. is created according to header file length and is received container, while receiving multiple file sizes, and representative has multiple same
Type file；
Step R3. inserts data content according to data number in header and block length (file size/total number packets amount)
Enter the corresponding position in container, so that it is guaranteed that file content sequence；
Step R4. verifies received content, if there is free information in container, represents data packetloss, then passes through step
The IP and source port number that rapid R1 is recorded are fed back, and the volume where file type, file size and container sky information is passed through
Number position, application transmitting terminal retransmit information corresponding to the number；
Step R5. repeats step R1 to step R4, and the information in container is corresponded to decode to lay equal stress on by file type and is write, is restored
File.
Compared with prior art, the invention has the following advantages:
1. the present invention use a kind of augmented reality method for exempting from sign-on ID, by dummy object model upload service end and by
Client dynamically load can also be real in the case where not updating client while the memory needed for reducing client application
The load and interaction of existing a variety of models；Solve traditional augmented reality to new scene bad adaptability, to mark rely on it is high with
And the demanding problem of exploitation.
2. the convolutional neural networks identification nicety of grading that training is completed is high, speed is fast, good generalization ability to know
Other object is also no longer limited to specific identifier.
3. sample data set quantitative requirement is being greatly lowered using the method training convolutional neural networks of transfer learning
Meanwhile the training time is also reduced, the more rapid adaptation application scenarios of network model are facilitated and classified service is provided.
4. substituting the mark tracking in traditional augmented reality method using the method for space reflection, the reality of mark can be got rid of
The limitation of border position combines virtual information more accurately, reasonably with real world.
5. being interacted in a manner of by conveniently gesture, voice and staring with model, user in terms of human-computer interaction
Operation is comfortable natural.
Detailed description of the invention
Fig. 1 is system architecture diagram of the invention；
Fig. 2 is client framework figure；
Fig. 3 is udp protocol header information figure；
Fig. 4 is server architecture figure；
Fig. 5 is 1 sample data set of embodiment；
Fig. 6 is 1 sample extending method of embodiment；
Fig. 7 is 1 model training result of example；
Fig. 8 is to implement 1 pair of 99 formula tank to take pictures；
Fig. 9 is the information transmission of embodiment 1；
Figure 10 is the information identification classification of embodiment 1；
Figure 11 is the dynamically load of the virtual information of embodiment 1.
Specific embodiment
Further technical solution of the present invention is clearly and completely described in the following with reference to the drawings and specific embodiments,
Obviously, described embodiment is only a part of the embodiments of the present invention, instead of all the embodiments.Based in the present invention
Embodiment, every other embodiment obtained by those of ordinary skill in the art without making creative efforts,
It shall fall within the protection scope of the present invention.
Embodiment 1:
As shown in Figure 1, one of the present embodiment exempts from the augmented reality method of sign-on ID, the augmented reality method is
Using the framework at client-server end, information transmission is carried out using udp protocol；Client provide human-computer interaction, information collection with
And the dynamic loading function of dummy model, server-side by the information received by transfer learning training convolutional neural networks into
Row identification classification, provides dummy model, to realize the effect of augmented reality.Human-computer interaction, the information collection of the client
And the dynamic loading function of dummy model includes information collection function, stares identification function, gesture identification function, speech recognition
Function, space reflection function and dynamic loading function.
The function realization for client, server-side and information transmission is described in detail separately below.
Client:
The architecture diagram of client is as shown in Fig. 2, the function of client is implemented by the following steps:
The information collection function of client described in step C1.；It is taken pictures using holographic glasses HoloLens to target, it will
Shooting result is saved with the format of Jpg；The picture that will take pictures is converted to Sprite format and is loaded onto provided by Uity3D engine
It is shown on UI carrier；Directly find out on UI taking pictures as a result, the photo bad to effect, takes pictures again to facilitate.
Client described in step C2. stares identification function；It is described stare identification be based on eye movement tracer technique, for
Track and selected holographic object, according to user's head position and direction by Unity3D engine Physical Raycast object
Ricoh's line obtains the feedback of collision result, the letter of position and colliding object including the point of impingement after colliding with holographic object
Breath, to realize the tracking of holographic object in scene and select；The application program of client stares identification function and shows virtual object
The selected and movement of body；
The gesture identification function of client described in step C3.；The gesture identification is to identify and track the position of user hand
With capture input gesture while state, system automatic trigger is fed back accordingly, to manipulate the virtual objects in scene；Input
Gesture includes Air-tap, Navigation gesture and tri- kinds of Bloom.
The speech identifying function of client described in step C4.；The speech recognition is user with the method for voice and holography
Object interaction, speech recognition are realized and keyword and corresponding feedback behavior is arranged in client application,
When user says keyword, client application responds preset feedback behavior；In the present embodiment, speech recognition and gesture
The concrete operations instruction of identification and respondent behavior are as shown in table 1.
1 speech recognition of table and the concrete operations instruction of gesture identification and respondent behavior
The space reflection functional development of client described in step C5.；The space reflection is by virtual world and real generation
Boundary is superimposed, is realized by following methods；
Step C5.1 is by using the holographic glasses HoloLens depth camera being equipped with and environment sensing camera, scanning
Environmental data and built-in triangulation around user obtain real world to realize the modelling and digitlization of real world
Digital physical space information；
Step C5.2 calculates whether digital physical space obtained above can place virtual hologram object in real time；By visitor
The space reflection function at family end, the spatial position of dummy model no longer by identification information in real world position constraint；
The dynamic loading function of client models described in step C6.；The dynamically load of the model is using holographic glasses
HoloLens loads the method for dummy model by access server to realize.The dynamically load of model is HoloLens wear-type
What augmented reality glasses were realized by access server load dummy model.The type of object is thousands of in daily life,
It is difficult for all objects to be placed in advance in augmented reality application program, and HoloLens is compared with high performance computer,
Its rendering capability, memory and performance are all extremely limited, so HoloLens can not carry a large amount of model for load.For
The problem is present invention employs the method for the dynamically load model from server, by Unity3D engine, by dummy model and
Script is packaged as the compressed package of AssetBundle, is uploaded to server, and HoloLens identified according to server-side after as a result,
Access server is downloaded the AssetBundle compressed package of corresponding model and is decompressed, thus the dynamically load of implementation model.
Information transmission:
The information transmission is to transmit the information of transmitting terminal, receiving end to information using udp protocol by processing
The information received is handled and restored；The information of transmission is carried out according to the maximum number of byte that its single can transmit first
Pretreatment, transmitting terminal according to file absolute path go obtain picture, then to picture carry out data encoding, data cutting and
The operation of header information is added, file type, file data length, number of data packets and data is added in header information and compiles
Number, receiving end according to data header information complete data decoding and recombination, and verify whether null data, if there is packet loss, high-ranking officers
It tests information back and applies for that transmitting terminal numbers packet loss information according to header data and retransmit.Udp protocol header information such as Fig. 3
It is shown.
The information processing of the transmitting terminal includes the following steps:
Step F1. encodes the information of transmission, and according to its information type, type content is encoded, and will compile
In the file type of the result insertion header of code；
Step F2. counts the result length after the information coding of transmission, and the resultant content of statistics is compiled
Code, will be in the file size of the result insertion header of coding；
Result equalization after the information coding of transmission is divided into multiple groups by step F3., and by the content of total number packets amount into
Row coding, will be in the number of data packets of the result insertion header of coding；
Data group after segmentation is numbered step F4. in order, and number content is encoded, the result of coding
It is inserted into the data number of header；
Step F5. repeats step F1 to step F4, pre-processes and retransmits in order to the information after segmentation, it is ensured that information is not
It a large amount of simultaneously can send.
The information processing of the receiving end includes the following steps:
Step R1. decodes the datagram header data received, by file type, IP and source port number classify, together
When create new thread and go to receive new information；
Step R2. is created according to header file length and is received container, while receiving multiple file sizes, and representative has multiple same
Type file；
Step R3. inserts data content according to data number in header and block length (file size/total number packets amount)
Enter the corresponding position in container, so that it is guaranteed that file content sequence；
Step R4. verifies received content, if there is free information in container, represents data packetloss, then passes through step
The IP and source port number that rapid R1 is recorded are fed back, and the volume where file type, file size and container sky information is passed through
Number position, application transmitting terminal retransmit information corresponding to the number；
Step R5. repeats step R1 to step R4, and the information in container is corresponded to decode to lay equal stress on by file type and is write, is restored
File.
Server-side:
The architecture diagram of server-side is as shown in Figure 4.The information received is passed through the volume of transfer learning training by the server-side
Product neural network carries out identification classification, provides dummy model；The present invention includes tank, panzer according to existing armoring class model
And the identifications such as fighting vehicle classification, and server is built using Apache, existing dummy model is uploaded on server.Institute
The convolutional neural networks for stating transfer learning training, are realized especially by following steps:
Step S1. establishes sample data set；Good sample data set is the basis of information Classification and Identification, and the present invention passes through
Internet approach has collected 15 class tanks and panzer image pattern, after arrangement and mark, obtains amounting to 1444 images
Sample, specimen types and distributed number are as shown in Figure 5.In order to over-fitting occur when mitigating and avoid model training, reduce
The influence of the recognition effect brought by Different categories of samples data bulk is unevenly distributed, the sample obtained by internet channels
Image is rotated by 90 ° by quantitative proportion, rotate 180 °, horizontal mirror image and vertical mirror image operation exptended sample data set, such as Fig. 6
(a)-(e) shown in.After expanding, data set total quantity reaches 9012, is finally made as tank armor identification
Data set；
Step S2. carries out model training on the data set for information identification that step S1 finally makes, from inhomogeneity
Each 75% sample image of selecting at random is training dataset in not, and remaining sample image is as test data set, iteration time
Number is 48 times, carries out model training；Training process are as follows:
Step S2.1 AlexNet network model carries out pre-training on the data set of ImageNet, will by the step
The parameter initialization of AlexNet network model；
Step S2.2 is since last three layers of AlexNet network model are configured as 1000 classification, so by last three
The full articulamentum of layer carries out re -training to adapt to new classification, and the parameter of new full articulamentum, Lai Shiying are retained by the step
The class categories of the data set for the tank armor identification that step S1 is established；
Step S2.3 joins first five layer of convolutional layer and its corresponding pond layer, the activation primitive and model in step S2.1
Several full articulamentums and its parameter in step S2.2 combine, and are finely adjusted to complete the training of model.
Step S3. is trained by the accuracy rate that penalty values, over-fitting ratio and test data are classified come scoring model
Effect；Wherein, shown in the accuracy rate such as formula (1) of test data classification
In formula (1), ExactQuantity indicates the correct quantity of test data classification results, and TotalQuantity is indicated
The total quantity of test data；The accuracy rate of test data classification is higher, indicates that the effect of network model classification is better.
Obtained by cross entropy loss function of the penalty values by Softmax, as shown in formula (2)
Wherein,1{yi=j } refer to indicative function, its value is 1 when value is true within " { } ", otherwise for
0；Penalty values indicate that the training result of network model is better closer to 0.
Shown in over-fitting ratio such as formula (3)
In formula, TrainAcc indicates training data accuracy rate, and TrainAcc is as shown in Equation 4
In formula, TrainExactQuantity is the correct quantity of training data classification results,
TrainTotalQuantity is the total quantity of training data.Over-fitting ratio indicates the extensive energy of network model closer to 1
Power is better.Network model is as shown in Figure 7 by transfer learning training training result.The accuracy rate average value of final test is
97.51%, and the over-fitting ratio of model is basically stable at 1.03 or so, and it is good to show that the network model of present invention training has
Good generalization ability.
Practical operation is carried out using system, HoloLens is connected into Wifi, with server-side according to IP address and port numbers
It realizes information transmission, starts test macro functions.HoloLens takes pictures result as shown in figure 8, Fig. 9 is service client information
Reception.As can be seen that picture packet loss phenomenon solves substantially by carrying out information checking in application layer from Fig. 8 and Fig. 9,
Service client information classification results are as shown in Figure 10, and Figure 11 is the dynamically load figure of virtual information.It can be seen from fig. 10 that this hair
The convolutional neural networks identification nicety of grading of bright middle transfer learning training is high, speed is fast, and higher generalization ability to identify
Object is also no longer limited to specific identifier, and 99 formula tank models are loaded out by Figure 11, and mobile by the method for space reflection
On to shelf (dotted line is tripod in Fig. 9).It can be seen from fig. 11 that using the method for space reflection for identifying tracking,
The physical location limitation of mark can be got rid of.C/S framework in through the invention, by dummy object model upload service end and by
Client dynamically load can also be in the case where not updating client while the memory needed for reducing client application
Realize the load and interaction of a variety of models.
Embodiment 2:
The difference of embodiment 2 and embodiment 1 is only that step S2.
Embodiment 2 is that each 70% sample image selected at random is training dataset, remaining sample from different classes of
For image as test data set, the number of iterations is 40 times, carries out model training；As a result consistent with embodiment 1.
Embodiment 3:
The difference of embodiment 3 and embodiment 1 is only that step S2.
Embodiment 3 is that each 80% sample image selected at random is training dataset, remaining sample from different classes of
For image as test data set, the number of iterations is 100 times, carries out model training；As a result consistent with embodiment 1.
This method is applicable in the augmented reality application of a large amount of dummy models, especially in engineering model field.
Although the present invention is described in detail referring to the foregoing embodiments, for those skilled in the art,
It is still possible to modify the technical solutions described in the foregoing embodiments, or part of technical characteristic is carried out etc.
With replacement, all within the spirits and principles of the present invention, any modification, equivalent replacement, improvement and so on should be included in this
Within the protection scope of invention.
Claims (10)
1. a kind of augmented reality method for exempting from sign-on ID, it is characterised in that: the augmented reality method is using client-clothes
The framework at business end carries out information transmission using udp protocol；Client provides human-computer interaction, information collection and dummy model
The information received is carried out identification classification by the convolutional neural networks of transfer learning training by dynamic loading function, server-side,
Dummy model is provided, to realize the effect of augmented reality.
2. a kind of augmented reality method for exempting from sign-on ID according to claim 1, it is characterised in that: the client
Human-computer interaction, information collection and dummy model dynamic loading function include information collection function, stare identification function, hand
Gesture identification function, speech identifying function, space reflection function and dynamic loading function.
3. a kind of augmented reality method for exempting from sign-on ID according to claim 2, it is characterised in that: the client
Information collection function, stare identification function, gesture identification function, speech identifying function, space reflection function and dynamically load
Function is realized using following steps:
The information collection function of client described in step C1.；It is taken pictures, will be shot to target using holographic glasses HoloLens
As a result it is saved with the format of Jpg；The picture that will take pictures is converted to Sprite format and is loaded onto the load of UI provided by Uity3D engine
It is shown on body；
Client described in step C2. stares identification function；It is described stare identification be based on eye movement tracer technique, for tracking and
Selected holographic object, according to user's head position and direction by Unity3D engine Physical Raycast physical light
Line, after being collided with holographic object, obtain collision result feedback, the information of position and colliding object including the point of impingement, from
And it realizes the tracking of holographic object in scene and selectes；
The gesture identification function of client described in step C3.；The gesture identification is position and the shape for identifying and tracking user hand
Capture input gesture while state, system automatic trigger is fed back accordingly, to manipulate the virtual objects in scene；
The speech identifying function of client described in step C4.；The speech recognition is user with the method and holographic object of voice
Interaction, speech recognition be by keyword being arranged in client application and corresponding feedback behavior and is realized, when with
When keyword is said at family, client application responds preset feedback behavior；
The space reflection functional development of client described in step C5.；The space reflection is by virtual world and real world phase
Superposition, is realized by following methods；
Step C5.1 scans user by using the holographic glasses HoloLens depth camera being equipped with and environment sensing camera
Around environmental data and built-in triangulation to realize the modelling and digitlization of real world obtain the number of real world
Word physical space information；
Step C5.2 calculates whether digital physical space obtained above can place virtual hologram object in real time；By client
Space reflection function, the spatial position of dummy model no longer by identification information in real world position constraint；
The dynamic loading function of client models described in step C6.；The dynamically load of the model is using holographic glasses
HoloLens loads the method for dummy model by access server to realize.
4. a kind of augmented reality method for exempting from sign-on ID according to claim 3, it is characterised in that: the step C3
In input gesture include Air-tap, Navigation gesture and tri- kinds of Bloom.
5. a kind of augmented reality method for exempting from sign-on ID according to claim 3, it is characterised in that: the step C6
The dummy model of middle server storage is in advance to be packaged as dummy model and script by Unity3D engine
The compressed package of AssetBundle is uploaded to server；Holographic glasses HoloLens identified according to server-side after as a result, access clothes
Business device is downloaded the AssetBundle compressed package of corresponding model and is decompressed, thus the dynamically load of implementation model.
6. a kind of augmented reality method for exempting from sign-on ID according to claim 1, it is characterised in that: the server-side
The information received is subjected to identification classification by the convolutional neural networks of transfer learning training, dummy model is provided；It is described to move
The convolutional neural networks for moving learning training, are realized especially by following steps:
Step S1. establishes sample data set；Good sample data set is the basis of information Classification and Identification, passes through internet channels
Sample image is obtained, sample image is rotated by 90 ° by quantitative proportion, rotates 180 °, horizontal mirror image and vertical mirror image operation
Exptended sample data set is finally made as the data set identified for information after expanding；
Step S2. carries out model training on the data set for information identification that step S1 finally makes, from different classes of
Each sample image for selecting 70%-80% at random is training dataset, and remaining sample image is as test data set, iteration time
Number is 40-100 times, carries out model training；
The effect that step S3. is trained by the accuracy rate that penalty values, over-fitting ratio and test data are classified come scoring model；
Wherein, shown in the accuracy rate such as formula (1) of test data classification
In formula (1), ExactQuantity indicates the correct quantity of test data classification results, and TotalQuantity indicates test
The total quantity of data；The accuracy rate of test data classification is higher, indicates that the effect of network model classification is better；
Obtained by cross entropy loss function of the penalty values by Softmax, as shown in formula (2)
Wherein,1{yi=j } refer to indicative function, otherwise it is 0 that when value is true in " { } ", its value, which is 1,；Damage
Mistake value indicates that the training result of network model is better closer to 0；
Shown in over-fitting ratio such as formula (3)
In formula, TrainAcc indicates training data accuracy rate, and TrainAcc is as shown in Equation 4
In formula, TrainExactQuantity is the correct quantity of training data classification results, and TrainTotalQuantity is
The total quantity of training data；Over-fitting ratio indicates that the generalization ability of network model is better closer to 1.
7. a kind of augmented reality method for exempting from sign-on ID according to claim 6, it is characterised in that: the step S2
Model training process are as follows:
Step S2.1 AlexNet network model carries out pre-training on the data set of ImageNet, will by the step
The parameter initialization of AlexNet network model；
Step S2.2 since last three layers of AlexNet network model are configured as 1000 classification, so will last three layers entirely
Articulamentum carries out re -training to adapt to new classification, and the parameter of new full articulamentum, Lai Shiying step are retained by the step
The class categories for the data set that S1 is established；
Step S2.3 by step S2.1 first five layer of convolutional layer and its corresponding pond layer, activation primitive and model parameter with
Full articulamentum and its parameter in step S2.2 combine, and are finely adjusted to complete the training of model.
8. a kind of augmented reality method for exempting from sign-on ID according to claim 1, it is characterised in that: the information passes
Defeated is to transmit the information of transmitting terminal to information using udp protocol by processing, and receiving end carries out the information received
Processing and reduction；The maximum number of byte that can be transmitted according to its single first pre-processes the information of transmission, in transmitting terminal root
It goes to obtain picture according to file absolute path, data encoding then is carried out to picture, data are cut and the behaviour of addition header information
Make, file type, file data length, number of data packets and data number is added in header information, receiving end is according to data
Header information completes data decoding and recombination, and verifies whether null data by verification information back and is applied sending out if there is packet loss
Sending end is numbered packet loss information according to header data and is retransmitted.
9. a kind of augmented reality method for exempting from sign-on ID according to claim 8, it is characterised in that: the transmitting terminal
Information processing include the following steps:
Step F1. encodes the information of transmission, and according to its information type, type content is encoded, and by coding
As a result it is inserted into the file type of header；
Step F2. counts the result length after the information coding of transmission, and the resultant content of statistics is encoded, will
In the file size of the result insertion header of coding；
Result equalization after the information coding of transmission is divided into multiple groups by step F3., and the content of total number packets amount is compiled
Code, will be in the number of data packets of the result insertion header of coding；
Data group after segmentation is numbered step F4. in order, and number content is encoded, the result insertion of coding
In the data number of header；
Step F5. repeats step F1 to step F4, pre-processes and retransmits in order to the information after segmentation, it is ensured that information will not be same
Shi great Liang is sent.
10. a kind of augmented reality method for exempting from sign-on ID according to claim 8, it is characterised in that: the reception
The information processing at end includes the following steps:
Step R1. decodes the datagram header data received, by file type, IP and source port number classify, create simultaneously
New thread is built to go to receive new information；
Step R2. is created according to header file length and is received container, while receiving multiple file sizes, and representative has multiple same types
File；
Data content is inserted into and holds according to data number in header and block length (file size/total number packets amount) by step R3.
Corresponding position in device, so that it is guaranteed that file content sequence；
Step R4. verifies received content, if there is free information in container, represents data packetloss, then passes through step R1
The IP and source port number recorded is fed back, and the number position where file type, file size and container sky information is passed through
It sets, application transmitting terminal retransmits information corresponding to the number；
Step R5. repeats step R1 to step R4, and the information in container is corresponded to decode to lay equal stress on by file type and is write, original text is gone back
Part.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910467466.5A CN110211240B (en) | 2019-05-31 | 2019-05-31 | Registration-free identification augmented reality method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910467466.5A CN110211240B (en) | 2019-05-31 | 2019-05-31 | Registration-free identification augmented reality method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110211240A true CN110211240A (en) | 2019-09-06 |
| CN110211240B CN110211240B (en) | 2022-10-21 |
Family
ID=67789867
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910467466.5A Active CN110211240B (en) | 2019-05-31 | 2019-05-31 | Registration-free identification augmented reality method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110211240B (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110782525A (en) * | 2019-11-08 | 2020-02-11 | 腾讯科技（深圳）有限公司 | Method, apparatus and medium for identifying virtual object in virtual environment |
| CN112486322A (en) * | 2020-12-07 | 2021-03-12 | 济南浪潮高新科技投资发展有限公司 | Multimodal AR (augmented reality) glasses interaction system based on voice recognition and gesture recognition |
| CN114205429A (en) * | 2021-12-14 | 2022-03-18 | 深圳壹账通智能科技有限公司 | Voice packet processing method, system, equipment and storage medium based on UDP protocol |
| CN117689508A (en) * | 2023-12-19 | 2024-03-12 | 杭州露电数字科技集团有限公司 | Intelligent teaching aid method and system based on MR equipment |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102739872A (en) * | 2012-07-13 | 2012-10-17 | 苏州梦想人软件科技有限公司 | Mobile terminal, and augmented reality method used for mobile terminal |
| CN106373198A (en) * | 2016-09-18 | 2017-02-01 | 福州大学 | Method for realizing augmented reality |
| CN106846237A (en) * | 2017-02-28 | 2017-06-13 | 山西辰涵影视文化传媒有限公司 | A kind of enhancing implementation method based on Unity3D |
| CN107015642A (en) * | 2017-03-13 | 2017-08-04 | 武汉秀宝软件有限公司 | A kind of method of data synchronization and system based on augmented reality |
| CN107331220A (en) * | 2017-09-01 | 2017-11-07 | 国网辽宁省电力有限公司锦州供电公司 | Transformer O＆M simulation training system and method based on augmented reality |
| CN107451661A (en) * | 2017-06-29 | 2017-12-08 | 西安电子科技大学 | A kind of neutral net transfer learning method based on virtual image data collection |
| US20180012411A1 (en) * | 2016-07-11 | 2018-01-11 | Gravity Jack, Inc. | Augmented Reality Methods and Devices |
| CN108416846A (en) * | 2018-03-16 | 2018-08-17 | 北京邮电大学 | It is a kind of without the three-dimensional registration algorithm of mark |
-
2019
- 2019-05-31 CN CN201910467466.5A patent/CN110211240B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102739872A (en) * | 2012-07-13 | 2012-10-17 | 苏州梦想人软件科技有限公司 | Mobile terminal, and augmented reality method used for mobile terminal |
| US20180012411A1 (en) * | 2016-07-11 | 2018-01-11 | Gravity Jack, Inc. | Augmented Reality Methods and Devices |
| CN106373198A (en) * | 2016-09-18 | 2017-02-01 | 福州大学 | Method for realizing augmented reality |
| CN106846237A (en) * | 2017-02-28 | 2017-06-13 | 山西辰涵影视文化传媒有限公司 | A kind of enhancing implementation method based on Unity3D |
| CN107015642A (en) * | 2017-03-13 | 2017-08-04 | 武汉秀宝软件有限公司 | A kind of method of data synchronization and system based on augmented reality |
| CN107451661A (en) * | 2017-06-29 | 2017-12-08 | 西安电子科技大学 | A kind of neutral net transfer learning method based on virtual image data collection |
| CN107331220A (en) * | 2017-09-01 | 2017-11-07 | 国网辽宁省电力有限公司锦州供电公司 | Transformer O＆M simulation training system and method based on augmented reality |
| CN108416846A (en) * | 2018-03-16 | 2018-08-17 | 北京邮电大学 | It is a kind of without the three-dimensional registration algorithm of mark |
Non-Patent Citations (4)
| Title |
|---|
| 宋亦凡: "基于移动端的无标识增强现实系统研究与开发", 《中国优秀硕士学位论文全文数据库 (信息科技辑)》 * |
| 张乐 等: "一种免注册标识的增强现实方法", 《科学技术与工程》 * |
| 张乐: "基于HoloLens的增强现实识别系统", 《中国优秀硕士学位论文全文数据库 (信息科技辑)》 * |
| 戚纯: "基于混合现实技术的数字博物馆应用研究", 《中国优秀硕士学位论文全文数据库 (信息科技辑)》 * |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110782525A (en) * | 2019-11-08 | 2020-02-11 | 腾讯科技（深圳）有限公司 | Method, apparatus and medium for identifying virtual object in virtual environment |
| CN112486322A (en) * | 2020-12-07 | 2021-03-12 | 济南浪潮高新科技投资发展有限公司 | Multimodal AR (augmented reality) glasses interaction system based on voice recognition and gesture recognition |
| CN114205429A (en) * | 2021-12-14 | 2022-03-18 | 深圳壹账通智能科技有限公司 | Voice packet processing method, system, equipment and storage medium based on UDP protocol |
| CN117689508A (en) * | 2023-12-19 | 2024-03-12 | 杭州露电数字科技集团有限公司 | Intelligent teaching aid method and system based on MR equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110211240B (en) | 2022-10-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110211240A (en) | A kind of augmented reality method for exempting from sign-on ID | |
| US10109101B2 (en) | Re-utilization of render assets for video compression | |
| CN110679152B (en) | Method and system for generating fused reality scene | |
| KR102318772B1 (en) | Domain Separation Neural Networks | |
| US10699471B2 (en) | Methods and systems for rendering frames based on a virtual entity description frame of a virtual scene | |
| Rapprich et al. | Application of modern technologies in popularization of the Czech volcanic geoheritage | |
| CN108764453B (en) | Modeling method and action prediction system for multi-agent synchronous game | |
| IL305340A (en) | High resolution neural rendering | |
| CN108236784A (en) | The training method and device of model, storage medium, electronic device | |
| CN113269858A (en) | Virtual scene rendering method and device, computer equipment and storage medium | |
| US11232623B2 (en) | Method and system for creating a neural net based lossy renderer | |
| CN117649478B (en) | Model training method, image processing method and electronic equipment | |
| KR102342738B1 (en) | Method and apparatus for generating mega size augmented reality image information | |
| US11544894B2 (en) | Latency-resilient cloud rendering | |
| CN117809084A (en) | Image recognition model training method, interaction method and device based on image recognition | |
| CN108520532A (en) | Identify the method and device of movement direction of object in video | |
| Meloni et al. | Messing up 3d virtual environments: Transferable adversarial 3d objects | |
| CN116561578A (en) | Sample generation method, device, equipment and storage medium | |
| CN110458164A (en) | Image processing method, device, equipment and computer readable storage medium | |
| CN116983660A (en) | Image processing method and related device | |
| CN116030040B (en) | Data processing method, device, equipment and medium | |
| CN115984094B (en) | Face safety generation method and equipment based on multi-loss constraint visual angle consistency | |
| CN117036652B (en) | Layout information generation method, model training method, device and electronic equipment | |
| US20230419655A1 (en) | Controlled image manipulations using semantic labels | |
| KR102667003B1 (en) | Method and system for training image generation model using content information |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |