CN109298780A - Information processing method, device, AR equipment and storage medium based on AR - Google Patents
Information processing method, device, AR equipment and storage medium based on AR Download PDFInfo
- Publication number
- CN109298780A CN109298780A CN201810974215.1A CN201810974215A CN109298780A CN 109298780 A CN109298780 A CN 109298780A CN 201810974215 A CN201810974215 A CN 201810974215A CN 109298780 A CN109298780 A CN 109298780A
- Authority
- CN
- China
- Prior art keywords
- user
- information
- target object
- image
- blinkpunkt
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/013—Eye tracking input arrangements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
- G06T19/006—Mixed reality
Abstract
The embodiment of the present invention provides a kind of information processing method based on AR, device, AR equipment and storage medium.The method of the embodiment of the present invention, by being used in AR device procedures in user, when user eyeball fixation object object, the image information for the target object that user eyeball is watched attentively can be got, according to the relevant information of the image information acquisition target object of target object, and the relevant information of target object is added in user's AR scene image within the vision, so as in real time according to user's blinkpunkt, any relevant information for searching the object that user is watched attentively is added in AR scene image, the virtual information being added in AR scene is greatly enriched；And the relevant information of user's object of interest can be shown to user according to the difference of user's focus, without binding corresponding virtual information for the fractional object in AR scene or AR scene, virtual information personalization in AR scene may be implemented and be superimposed.
Description
Technical field
The present embodiments relate to augmented reality field more particularly to a kind of information processing methods based on AR, dress
It sets, AR equipment and storage medium.
Background technique
Augmented reality (Augmented Reality, abbreviation AR) is a kind of position for calculating camera image in real time
Set and angle and plus respective image, video, 3D model technology, the target of this technology is on the screen virtual world set
In real world and interacted.
AR equipment is one important application of augmented reality, can be mutual by the information of virtual information and real world
Virtual information is presented in the AR scene of true environment composition by superposition.But current AR equipment, establishing AR equipment
When model, corresponding virtual information is bound for the fractional object in AR scene or AR scene in advance.When using AR equipment, only
Can by advance in AR scene or AR scene object binding fixation virtual information be added in AR scene, display it is virtual
Information is single, and can not provide the relevant information of the object of unbound virtual information.
Summary of the invention
The embodiment of the present invention provides a kind of information processing method based on AR, device, AR equipment and storage medium, to solve
Certainly current AR equipment when in use, can only be by the fixation virtual information in advance with the object binding in AR scene or AR scene
It being added in AR scene, the virtual information of display is single, and the relevant information of the object of unbound virtual information can not be provided
Problem.
The one aspect of the embodiment of the present invention is to provide a kind of information processing method based on AR, comprising:
In user using in AR device procedures, the location information of the blinkpunkt of the user eyeball is obtained；
According to the location information of the blinkpunkt of the user eyeball, the corresponding target of blinkpunkt of the user eyeball is obtained
The image information of object；
According to the image information of the target object, the relevant information of the target object is obtained；
The relevant information of the target object is added in user AR scene image within the vision.
The other side of the embodiment of the present invention is to provide a kind of information processing unit based on AR, comprising:
Eye tracking module, for, using in AR device procedures, obtaining the position of the blinkpunkt of the user eyeball in user
Confidence breath；
Image interception module obtains the user eyeball for the location information according to the blinkpunkt of the user eyeball
The corresponding target object of blinkpunkt image information；
Data obtaining module obtains the related letter of the target object for the image information according to the target object
Breath；
Superposition processing module, for the relevant information of the target object to be added to user AR within the vision
In scene image.
The other side of the embodiment of the present invention is to provide a kind of AR equipment, comprising:
Memory, processor, and it is stored in the computer journey that can be run on the memory and on the processor
Sequence,
The processor realizes method described above when running the computer program.
The other side of the embodiment of the present invention is to provide a kind of computer readable storage medium, is stored with computer journey
Sequence,
The computer program realizes method described above when being executed by processor.
Information processing method based on AR, device, AR equipment and storage medium provided in an embodiment of the present invention, by with
Family, when user eyeball fixation object object, can get the object that user eyeball is watched attentively using in AR device procedures
The image information of body, according to the relevant information of the image information acquisition target object of target object, and by the correlation of target object
Information superposition is into user's AR scene image within the vision, so as to according to user's blinkpunkt, search in real time
Any relevant information for the object that user is watched attentively is added in AR scene image, is greatly enriched and is added in AR scene
Virtual information；And the relevant information of user's object of interest, nothing can be shown to user according to the difference of user's focus
Corresponding virtual information need to be bound for the fractional object in AR scene or AR scene, virtual information in AR scene may be implemented
Propertyization superposition.
Detailed description of the invention
Fig. 1 is the information processing method flow chart based on AR that the embodiment of the present invention one provides；
Fig. 2 is the information processing method flow chart provided by Embodiment 2 of the present invention based on AR；
Fig. 3 is the structural schematic diagram for the information processing unit based on AR that the embodiment of the present invention three provides；
Fig. 4 is the structural schematic diagram for the AR equipment that the embodiment of the present invention five provides.
Through the above attached drawings, it has been shown that the specific embodiment of the present invention will be hereinafter described in more detail.These attached drawings
It is not intended to limit the range of design of the embodiment of the present invention in any manner with verbal description, but by reference to specific reality
Applying example is that those skilled in the art illustrate idea of the invention.
Specific embodiment
Example embodiments are described in detail here, and the example is illustrated in the accompanying drawings.Following description is related to
When attached drawing, unless otherwise indicated, the same numbers in different drawings indicate the same or similar elements.Following exemplary embodiment
Described in embodiment do not represent all embodiments consistent with the embodiment of the present invention.On the contrary, they be only with
The example of the consistent device and method of as detailed in the attached claim, the embodiment of the present invention some aspects.
Term " first " involved in the embodiment of the present invention, " second " etc. are used for description purposes only, and should not be understood as referring to
Show or imply relative importance or implicitly indicates the quantity of indicated technical characteristic.In the description of following embodiment,
The meaning of " plurality " is two or more, unless otherwise specifically defined.
These specific embodiments can be combined with each other below, may be at certain for the same or similar concept or process
It is repeated no more in a little embodiments.Below in conjunction with attached drawing, the embodiment of the present invention is described.
Embodiment one
Fig. 1 is the information processing method flow chart based on AR that the embodiment of the present invention one provides.The embodiment of the present invention is directed to
Current AR equipment when in use, can only fold the fixation virtual information in advance with the object binding in AR scene or AR scene
It is added in AR scene, the virtual information of display is single, and the relevant information that can not provide the object of unbound virtual information is asked
Topic, provides the information processing method based on AR.Method in the present embodiment is applied to AR equipment, for example, it may be AR glasses,
AR helmet etc..The executing subject of this method can be the processor of AR equipment.As shown in Figure 1, specific step is as follows for this method:
Step S101, the location information of the blinkpunkt of user eyeball is obtained using in AR device procedures in user.
When user uses AR equipment, when user watches a target object attentively, the eyeball of user can stop a position
A period of time is stayed, therefore by the motion state of tracking user eyeball, can determine the location information of the blinkpunkt of user eyeball,
It may thereby determine that whether user watches a target object attentively.
In the present embodiment, the eye tracking device of the exercise data of the eyeball for acquiring user is provided in AR equipment,
The exercise data of the eyeball of the user of acquisition can be sent to the processor of AR equipment by eye tracking device.The processing of AR equipment
The blinkpunkt of user eyeball is calculated according to the exercise data of the eyeball of user in device.
Optionally, eye tracking device can be the photographic device of the eyes image information for acquiring user, and being used for will
The eyes image information of the user of acquisition is sent to the processor of AR equipment, and the processing of AR equipment can use image recognition technology
It identifies user eyeball, calculates the motion profile of user eyeball；And an object is watched attentively determining that user eyeball stops
When, calculate the blinkpunkt of user eyeball.
Optionally, eye tracking device can also be that eye tracker etc. can be used in tracking the motion state of eyeball and obtain eye
The location information of determining blinkpunkt is sent to the processor of AR equipment by the device of ball blinkpunkt, eye tracking device.
In user using in AR device procedures, the location information of the blinkpunkt of user eyeball is obtained, it specifically can be using such as
Under type is realized:
In user using in AR device procedures, the processor of AR equipment receives what eye tracking device was acquired and sent in real time
The location information of the blinkpunkt of the eyeball of user；According to the location information of the blinkpunkt of the eyeball of user, user eyeball is determined
Position of the blinkpunkt in AR scene image.
Specifically, the processor of AR equipment can be according to pair of the coordinate system of the coordinate system and AR equipment of eye tracking device
It should be related to, the location information of the blinkpunkt received is transformed under the coordinate system of AR equipment, obtain the blinkpunkt of user eyeball
Position in AR scene image.
Step S102, according to the location information of the blinkpunkt of user eyeball, the corresponding mesh of blinkpunkt of user eyeball is obtained
Mark the image information of object.
It, can be further according to the blinkpunkt of user eyeball after the location information for the blinkpunkt for getting user eyeball
The image information for obtaining the corresponding target object of blinkpunkt that is to say the image letter for obtaining the target object of user's current fixation
Breath.
Step S103, according to the image information of target object, the relevant information of target object is obtained.
Wherein, the relevant information of target object may include: any information extracted from the image information of target object,
It is further according to the information extracted from the image information of target object according to the information that the image information of target object searches
Any information relevant to target object such as the information searched.
The relevant information of target object can be text information, pictorial information, any in video information and voice messaging
A kind of information, or it is also possible to the combination of much information.
After getting the image information of target object of user's current fixation, the relevant information of target object is obtained
A kind of feasible embodiment are as follows: scanned for according to the image information to target object by search engine, obtain object
The relevant information of body.
Optionally, the feasible embodiment of another kind of the relevant information of target object is obtained are as follows: by target object
Image information carry out image recognition, the relevant information of target object is extracted from image information.
Optionally, the feasible embodiment of another kind of the relevant information of target object is obtained are as follows: by target object
Image information carry out image recognition, the essential information of target object is extracted from image information；According to the target extracted
The essential information of object is scanned for by search engine, obtains the relevant information of target object.
Optionally, the feasible embodiment of another kind of the relevant information of target object is obtained are as follows: by target object
Image information carry out image recognition, the attribute informations such as type, the title of target object are extracted from image information；According to mentioning
The attribute information of the target object of taking-up is scanned for by search engine, obtains the relevant information of target object.
Step S104, the relevant information of target object is added in user's AR scene image within the vision.
After getting the relevant information of target object, the relevant information superposition of target object is shown to the user visual field
In AR scene image in range near target object.
Optionally, the relevant information of target object can be superimposed and is shown in user's AR scene image within the vision
Target object where region.
Optionally, the relevant information of target object can be superimposed and is shown in user's AR scene image within the vision
Predeterminable area.The relevant information of target object is added to user's AR scene within the vision for example, can preset
In the region of the default size at top, left side, right side or bottom in image.
The embodiment of the present invention, can when user eyeball fixation object object by using in AR device procedures in user
The image information for getting the target object that user eyeball is watched attentively, according to the image information acquisition target object of target object
Relevant information, and the relevant information of target object is added in user's AR scene image within the vision, so as to reality
When according to user's blinkpunkt, any relevant information for searching the object that user is watched attentively is added in AR scene image,
It is greatly enriched the virtual information being added in AR scene；And it can be shown according to the difference of user's focus to user
The relevant information of user's object of interest, without binding corresponding virtual letter for the fractional object in AR scene or AR scene
Breath may be implemented virtual information personalization in AR scene and be superimposed.
Embodiment two
Fig. 2 is the information processing method flow chart provided by Embodiment 2 of the present invention based on AR.In above-described embodiment one
On the basis of, in the present embodiment, in user using in AR device procedures, obtain the location information of the blinkpunkt of user eyeball, comprising:
In user using in AR device procedures, the eyes image information of user is obtained in real time；According to the eyes image information of user, know
User eyeball and the motion state of user eyeball Chu not be tracked；If user eyeball residence time is greater than preset duration, it is determined that
Position of the blinkpunkt of user eyeball in AR scene image.According to the image information of target object, the phase of target object is obtained
Close information, comprising: the image information of target object is sent to search server, so that search server search and feedback target
The relevant information of object；Receive the relevant information of the target object of search server feedback.As shown in Fig. 2, this method specifically walks
It is rapid as follows:
Step S201, the eyes image information of user is obtained in real time using in AR device procedures in user.
In user using in AR device procedures, when user watches a target object attentively, the eyeball of user can be a position
It sets and stays for some time, therefore by the motion state of tracking user eyeball, can determine whether user watches an object attentively
Body.
In the present embodiment, the eye tracking device of the exercise data of the eyeball for acquiring user is provided in AR equipment,
The exercise data of the eyeball of the user of acquisition can be sent to the processor of AR equipment by eye tracking device, by AR equipment
Device is managed according to the exercise data of the eyeball of user, the blinkpunkt of user eyeball is calculated.
Specifically, eye tracking device can be the photographic device of the eyes image information for acquiring user, being used for will
The eyes image information of the user of acquisition is sent to the processor of AR equipment.
Step S202, it according to the eyes image information of user, identifies user eyeball and tracks the movement shape of user eyeball
State.
After getting the eyes image information of user, the processing of AR equipment can be identified using image recognition technology
User eyeball calculates the motion profile of user eyeball；And judging that user eyeball is in moving condition or resting state.
Optionally, it if the time that the eyeball of user remains stationary is greater than preset duration, can determine at the eyeball of user
In resting state；If the time that the eyeball of user remains stationary is less than or equal to preset duration, the eye of user can be determined
Ball is in moving condition.
Wherein, the value range of preset duration can be 300 milliseconds to 500 milliseconds.Preset duration can be by technical staff
It is set according to actual needs, the present embodiment is not specifically limited herein.
If step S203, user eyeball residence time is greater than preset duration, it is determined that the blinkpunkt of user eyeball is in AR
Position in scene image.
If user eyeball is greater than preset duration in some position residence time, it is determined that the eyeball of user, which is in, to stop
State can calculate position of the blinkpunkt of user eyeball in AR scene image.
Above-mentioned steps S201-S203 is to track the motion state of the eyeball of user when user uses AR equipment, obtain and use
A kind of feasible embodiment of the blinkpunkt of family eyeball.
Step S204, the image information of the corresponding target object of blinkpunkt of user eyeball is obtained.
In the blinkpunkt for getting user eyeball, position of the blinkpunkt of determining user eyeball in AR scene image that is to say
After setting, the blinkpunkt that can further get user eyeball corresponds to the image letter of the target object in AR scene image
Breath.
In the present embodiment, a kind of feasible reality of the image information of the corresponding target object of blinkpunkt of user eyeball is obtained
Apply mode are as follows:
According to position of the blinkpunkt of user eyeball in AR scene image, obtains in AR scene image with blinkpunkt and be
The image information of the default size of the heart.
Wherein, default size can be set by technical staff according to actual scene, and the present embodiment is not done specifically herein
It limits.
Optionally, the another kind for obtaining the image information of the corresponding target object of blinkpunkt of user eyeball is a kind of feasible
Embodiment are as follows:
According to position of the blinkpunkt of user eyeball in AR scene image, AR scene image is carried out at image segmentation
Reason, obtains the image information of the cut zone in AR scene image where blinkpunkt.
For example, AR scene image is usually depth image, according to position of the blinkpunkt of user eyeball in AR scene image
It sets, can will come out, obtain with the consistent region segmentation of the depth information of the blinkpunkt around blinkpunkt in AR scene image
The image information of cut zone where the blinkpunkt.
Step S205, the image information of target object is sent to search server so that search server search and it is anti-
Present the relevant information of target object.
Wherein, the relevant information of target object may include: any information extracted from the image information of target object,
It is further according to the information extracted from the image information of target object according to the information that the image information of target object searches
Any information relevant to target object such as the information searched.
The relevant information of target object can be text information, pictorial information, any in video information and voice messaging
A kind of information, or it is also possible to the combination of much information.
In the present embodiment, the relevant information of target object can be searched for by search server.Specifically, getting mesh
After the image information for marking object, the image information of target object is sent to search server by the processor of AR equipment.Search
Server is scanned for according to the image information of target object, and search result is fed back to the processor of AR equipment.
Wherein, search server can be any one search engine in the prior art, and the present embodiment is not done specifically herein
It limits.
For example, when user watches one plant of plant attentively by AR equipment, the available picture to the plant of AR equipment, concurrently
Give search engine.Search engine is scanned for by the picture to the plant, the title of the available plant, affiliated point
Class, growing environment, implantation methods etc. text information can also all be the pictorial informations etc. to the plant in different growth periods
Relevant information, and the relevant information of the plant is sent to AR equipment.
For example, when user watches a famous people attentively by AR equipment, the available picture to this people of AR equipment,
And it is sent to search engine.Search engine is scanned for by the picture to this people, the name of this available people, letter
The text informations such as Jie, resume, prominent deed and the picture of this people, video etc., and the relevant information of this people is sent
Give AR equipment.
For example, when user watches a commodity attentively by AR equipment, the available picture to this part commodity of AR equipment, and
It is sent to search engine.Search engine is scanned for by the picture to this part commodity, the title of this available part commodity, valence
Lattice, manufacturer, shelf-life, advertisement video etc. relevant information, and the relevant information of this commodity is sent to AR equipment.
Optionally, the processor of AR equipment can also carry out image recognition by the image information to target object, from figure
Essential information as extracting target object in information；The essential information of the target object extracted is sent to search service
Device.Search server is scanned for according to the essential information of target object, and search result is fed back to the processor of AR equipment.
Optionally, the processor of AR equipment can also carry out image recognition by the image information to target object, from figure
As attribute informations such as the type, the titles that extract target object in information；The attribute information of the target object extracted is sent
To search server.Search server is scanned for according to the attribute information of target object, and search result is fed back to AR equipment
Processor.
Step S206, the relevant information of the target object of search server feedback is received.
The processor of AR equipment receives the search result of search server feedback, and using the search result received as mesh
Mark the relevant information of object.
Since the search result of search server includes a large amount of information, usual search server can be according in search result
Information and the correlation of target object are ranked up a plurality of information in search result.Optionally, search server is received
After the relevant information of the target object of feedback, the processor of AR equipment can also be to the target object that search server is fed back
Relevant information is filtered, and retains the information of the wherein highest preset quantity of correlation.
Wherein, preset quantity can be set according to actual needs by technical staff, for example, preset quantity can be 1,
3,5,8,10 etc., the present embodiment is not specifically limited herein.
Above-mentioned steps S204-S205 is the one kind for obtaining the image information of the corresponding target object of blinkpunkt of user eyeball
Feasible embodiment.
Step S207, the relevant information of target object is added in user's AR scene image within the vision.
After getting the relevant information of target object, the relevant information superposition of target object is shown to the user visual field
In AR scene image in range near target object.
In the present embodiment, the relevant information of target object is added in user's AR scene image within the vision, is had
Body can be realized in the following way:
According to position of the blinkpunkt of user eyeball in AR scene image, the relevant information of target object is added to AR
Less than in the region of pre-determined distance at a distance from blinkpunkt in scene image.
Wherein, pre-determined distance can be set according to actual needs by technical staff, and the present embodiment is not done specifically herein
It limits.
Optionally, the relevant information of target object is added in user's AR scene image within the vision, specifically may be used
To realize in the following way:
Generate the corresponding pictorial information of relevant information, animation information or the voice messaging of target object；By target object
The corresponding pictorial information of relevant information, animation information or voice messaging, the user that is added to AR scene figure within the vision
As in.
For example, AR equipment can incite somebody to action if the relevant information of target object includes the related text description information of a people
Related text description information is converted to corresponding voice, is added to user's AR scene within the vision in the form that voice plays
In image.For example, can be played with voice " this people that you are currently seen be Zhao so-and-so, the age is ... ".
Optionally, the relevant information of target object can be superimposed and is shown in user's AR scene image within the vision
Target object where region.
Optionally, the relevant information of target object can be superimposed and is shown in user's AR scene image within the vision
Predeterminable area.The relevant information of target object is added to user's AR scene within the vision for example, can preset
In the region of the default size at top, left side, right side or bottom in image.
The embodiment of the present invention, can when user eyeball fixation object object by using in AR device procedures in user
The image information for getting the target object that user eyeball is watched attentively, according to the image information acquisition target object of target object
Relevant information, and the relevant information of target object is added in user's AR scene image within the vision, so as to reality
When according to user's blinkpunkt, any relevant information for searching the object that user is watched attentively is added in AR scene image,
It is greatly enriched the virtual information being added in AR scene；And it can be shown according to the difference of user's focus to user
The relevant information of user's object of interest, without binding corresponding virtual letter for the fractional object in AR scene or AR scene
Breath may be implemented virtual information personalization in AR scene and be superimposed.
Embodiment three
Fig. 3 is the structural schematic diagram for the information processing unit based on AR that the embodiment of the present invention three provides.The present invention is implemented
The information processing unit based on AR that example provides can execute the process flow of the offer of the information processing method embodiment based on AR.
As shown in figure 3, the device 30 includes: eye tracking module 301, image interception module 302, data obtaining module 303 and superposition
Processing module 304.
Specifically, eye tracking module 301 is used to obtain the blinkpunkt of user eyeball using in AR device procedures in user
Location information.
Image interception module 302 is used for the location information of the blinkpunkt according to user eyeball, obtains watching attentively for user eyeball
The image information of the corresponding target object of point.
Data obtaining module 303 is used for the image information according to target object, obtains the relevant information of target object.
Superposition processing module 304 is used for user's AR scene figure within the vision that the relevant information of target object is added to
As in.
Device provided in an embodiment of the present invention can be specifically used for executing embodiment of the method provided by above-described embodiment one,
Details are not described herein again for concrete function.
The embodiment of the present invention, can when user eyeball fixation object object by using in AR device procedures in user
The image information for getting the target object that user eyeball is watched attentively, according to the image information acquisition target object of target object
Relevant information, and the relevant information of target object is added in user's AR scene image within the vision, so as to reality
When according to user's blinkpunkt, any relevant information for searching the object that user is watched attentively is added in AR scene image,
It is greatly enriched the virtual information being added in AR scene；And it can be shown according to the difference of user's focus to user
The relevant information of user's object of interest, without binding corresponding virtual letter for the fractional object in AR scene or AR scene
Breath may be implemented virtual information personalization in AR scene and be superimposed.
Example IV
On the basis of above-described embodiment three, in the present embodiment, the eye tracking module is also used to:
In user using in AR device procedures, the eyes image information of user is obtained in real time；According to the eye figure of user
As information, identifies user eyeball and track the motion state of user eyeball；If user eyeball residence time is greater than default
It is long, it is determined that position of the blinkpunkt of user eyeball in AR scene image.
Optionally, eye tracking module is also used to:
In user using in AR device procedures, the note of the eyeball for the user that eye tracking device is acquired and sent in real time is received
The location information of viewpoint；According to the location information of the blinkpunkt of the eyeball of user, determine the blinkpunkt of user eyeball in AR scene
Position in image.
Optionally, image interception module is also used to:
According to position of the blinkpunkt of user eyeball in AR scene image, obtains in AR scene image with blinkpunkt and be
The image information of the default size of the heart.
Optionally, image interception module is also used to:
According to position of the blinkpunkt of user eyeball in AR scene image, AR scene image is carried out at image segmentation
Reason, obtains the image information of the cut zone in AR scene image where blinkpunkt.
Optionally, data obtaining module is also used to:
The image information of target object is sent to search server, so that search server search and feedback target object
Relevant information；Receive the relevant information of the target object of search server feedback.
Optionally, data obtaining module is also used to:
The relevant information of the target object of search server feedback is filtered, it is highest default to retain wherein correlation
The information of quantity.
Optionally, superposition processing module is also used to:
According to position of the blinkpunkt of user eyeball in AR scene image, the relevant information of target object is added to AR
Less than in the region of pre-determined distance at a distance from blinkpunkt in scene image.
Optionally, superposition processing module is also used to:
Generate the corresponding pictorial information of relevant information, animation information or the voice messaging of target object；By target object
The corresponding pictorial information of relevant information, animation information or voice messaging, the user that is added to AR scene figure within the vision
As in.
Device provided in an embodiment of the present invention can be specifically used for executing embodiment of the method provided by above-described embodiment two,
Details are not described herein again for concrete function.
The embodiment of the present invention, can when user eyeball fixation object object by using in AR device procedures in user
The image information for getting the target object that user eyeball is watched attentively, according to the image information acquisition target object of target object
Relevant information, and the relevant information of target object is added in user's AR scene image within the vision, so as to reality
When according to user's blinkpunkt, any relevant information for searching the object that user is watched attentively is added in AR scene image,
It is greatly enriched the virtual information being added in AR scene；And it can be shown according to the difference of user's focus to user
The relevant information of user's object of interest, without binding corresponding virtual letter for the fractional object in AR scene or AR scene
Breath may be implemented virtual information personalization in AR scene and be superimposed.
Embodiment five
Fig. 4 is the structural schematic diagram for the AR equipment that the embodiment of the present invention five provides.As shown in figure 4, the AR equipment 40 includes:
Processor 401, memory 402, and it is stored in the computer program that can be executed on memory 402 and by processor 401.
Processor 401 realizes any of the above-described embodiment of the method when executing and storing in the computer program on memory 402
The information processing method based on AR provided.
The embodiment of the present invention, can when user eyeball fixation object object by using in AR device procedures in user
The image information for getting the target object that user eyeball is watched attentively, according to the image information acquisition target object of target object
Relevant information, and the relevant information of target object is added in user's AR scene image within the vision, so as to reality
When according to user's blinkpunkt, any relevant information for searching the object that user is watched attentively is added in AR scene image,
It is greatly enriched the virtual information being added in AR scene；And it can be shown according to the difference of user's focus to user
The relevant information of user's object of interest, without binding corresponding virtual letter for the fractional object in AR scene or AR scene
Breath may be implemented virtual information personalization in AR scene and be superimposed.
In addition, the embodiment of the present invention also provides a kind of computer readable storage medium, it is stored with computer program, the meter
Calculation machine program realizes the information processing method based on AR that any of the above-described embodiment of the method provides when being executed by processor.
In several embodiments provided by the present invention, it should be understood that disclosed device and method can pass through it
Its mode is realized.For example, the apparatus embodiments described above are merely exemplary, for example, the division of the unit, only
Only a kind of logical function partition, there may be another division manner in actual implementation, such as multiple units or components can be tied
Another system is closed or is desirably integrated into, or some features can be ignored or not executed.Another point, it is shown or discussed
Mutual coupling, direct-coupling or communication connection can be through some interfaces, the INDIRECT COUPLING or logical of device or unit
Letter connection can be electrical property, mechanical or other forms.
The unit as illustrated by the separation member may or may not be physically separated, aobvious as unit
The component shown may or may not be physical unit, it can and it is in one place, or may be distributed over multiple
In network unit.It can select some or all of unit therein according to the actual needs to realize the mesh of this embodiment scheme
's.
It, can also be in addition, the functional units in various embodiments of the present invention may be integrated into one processing unit
It is that each unit physically exists alone, can also be integrated in one unit with two or more units.Above-mentioned integrated list
Member both can take the form of hardware realization, can also realize in the form of hardware adds SFU software functional unit.
The above-mentioned integrated unit being realized in the form of SFU software functional unit can store and computer-readable deposit at one
In storage media.Above-mentioned SFU software functional unit is stored in a storage medium, including some instructions are used so that a computer
It is each that equipment (can be personal computer, server or the network equipment etc.) or processor (processor) execute the present invention
The part steps of embodiment the method.And storage medium above-mentioned includes: USB flash disk, mobile hard disk, read-only memory (Read-
Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic or disk etc. it is various
It can store the medium of program code.
Those skilled in the art can be understood that, for convenience and simplicity of description, only with above-mentioned each functional module
Division progress for example, in practical application, can according to need and above-mentioned function distribution is complete by different functional modules
At the internal structure of device being divided into different functional modules, to complete all or part of the functions described above.On
The specific work process for stating the device of description, can refer to corresponding processes in the foregoing method embodiment, and details are not described herein.
Those skilled in the art after considering the specification and implementing the invention disclosed here, will readily occur to of the invention its
Its embodiment.The present invention is directed to cover any variations, uses, or adaptations of the invention, these modifications, purposes or
Person's adaptive change follows general principle of the invention and including the undocumented common knowledge in the art of the present invention
Or conventional techniques.The description and examples are only to be considered as illustrative, and true scope and spirit of the invention are by following
Claims are pointed out.
It should be understood that the present invention is not limited to the precise structure already described above and shown in the accompanying drawings, and
And various modifications and changes may be made without departing from the scope thereof.The scope of the present invention is only limited by appended claims
System.
Claims (20)
1. a kind of information processing method based on AR characterized by comprising
In user using in AR device procedures, the location information of the blinkpunkt of the user eyeball is obtained；
According to the location information of the blinkpunkt of the user eyeball, the corresponding target object of blinkpunkt of the user eyeball is obtained
Image information；
According to the image information of the target object, the relevant information of the target object is obtained；
The relevant information of the target object is added in user AR scene image within the vision.
2. the method according to claim 1, wherein described use in AR device procedures in user, described in acquisition
The location information of the blinkpunkt of user eyeball, comprising:
In user using in AR device procedures, the eyes image information of the user is obtained in real time；
According to the eyes image information of the user, identifies the user eyeball and track the movement shape of the user eyeball
State；
If the user eyeball residence time is greater than preset duration, it is determined that the blinkpunkt of the user eyeball is AR described
Position in scape image.
3. the method according to claim 1, wherein described use in AR device procedures in user, described in acquisition
The location information of the blinkpunkt of user eyeball, comprising:
In user using in AR device procedures, the note of the eyeball for the user that eye tracking device is acquired and sent in real time is received
The location information of viewpoint；
According to the location information of the blinkpunkt of the eyeball of the user, determine the blinkpunkt of the user eyeball in the AR scene
Position in image.
4. according to the method in claim 2 or 3, which is characterized in that the position of the blinkpunkt according to the user eyeball
Confidence breath, obtains the image information of the corresponding target object of blinkpunkt of the user eyeball, comprising:
According to position of the blinkpunkt of the user eyeball in the AR scene image, obtain in the AR scene image with institute
State the image information of the default size centered on blinkpunkt.
5. according to the method in claim 2 or 3, which is characterized in that the position of the blinkpunkt according to the user eyeball
Confidence breath, obtains the image information of the corresponding target object of blinkpunkt of the user eyeball, comprising:
According to position of the blinkpunkt of the user eyeball in the AR scene image, image is carried out to the AR scene image
Dividing processing obtains the image information of the cut zone where blinkpunkt described in the AR scene image.
6. the method according to claim 1, wherein the image information according to the target object, obtains
The relevant information of the target object, comprising:
The image information of the target object is sent to search server, so that described in described search server search and feedback
The relevant information of target object；
Receive the relevant information of the target object of described search server feedback.
7. according to the method described in claim 6, it is characterized in that, the target for receiving described search server feedback
After the relevant information of object, further includes:
The relevant information of the target object of described search server feedback is filtered, it is highest to retain wherein correlation
The information of preset quantity.
8. the institute the method according to claim 1, wherein the relevant information by the target object is added to
It states in user's AR scene image within the vision, comprising:
According to position of the blinkpunkt of the user eyeball in the AR scene image, by the relevant information of the target object
It is added in the AR scene image at a distance from the blinkpunkt less than in the region of pre-determined distance.
9. according to the method described in claim 8, the institute it is characterized in that, the relevant information by the target object is added to
It states in user's AR scene image within the vision, comprising:
Generate the corresponding pictorial information of relevant information, animation information or the voice messaging of the target object；
By the corresponding pictorial information of the relevant information of the target object, animation information or voice messaging, be added to the use
In the AR scene image within the vision of family.
10. a kind of information processing unit based on AR characterized by comprising
Eye tracking module, for using the position letter in AR device procedures, obtaining the blinkpunkt of the user eyeball in user
Breath；
Image interception module obtains the note of the user eyeball for the location information according to the blinkpunkt of the user eyeball
The image information of the corresponding target object of viewpoint；
Data obtaining module obtains the relevant information of the target object for the image information according to the target object；
Superposition processing module, for the relevant information of the target object to be added to user AR scene within the vision
In image.
11. device according to claim 10, which is characterized in that the eye tracking module is also used to:
In user using in AR device procedures, the eyes image information of the user is obtained in real time；
According to the eyes image information of the user, identifies the user eyeball and track the movement shape of the user eyeball
State；
If the user eyeball residence time is greater than preset duration, it is determined that the blinkpunkt of the user eyeball is AR described
Position in scape image.
12. device according to claim 10, which is characterized in that the eye tracking module is also used to:
In user using in AR device procedures, the note of the eyeball for the user that eye tracking device is acquired and sent in real time is received
The location information of viewpoint；
According to the location information of the blinkpunkt of the eyeball of the user, determine the blinkpunkt of the user eyeball in the AR scene
Position in image.
13. device according to claim 11 or 12, which is characterized in that described image interception module is also used to:
According to position of the blinkpunkt of the user eyeball in the AR scene image, obtain in the AR scene image with institute
State the image information of the default size centered on blinkpunkt.
14. device according to claim 11 or 12, which is characterized in that described image interception module is also used to:
According to position of the blinkpunkt of the user eyeball in the AR scene image, image is carried out to the AR scene image
Dividing processing obtains the image information of the cut zone where blinkpunkt described in the AR scene image.
15. device according to claim 10, which is characterized in that the data obtaining module is also used to:
The image information of the target object is sent to search server, so that described in described search server search and feedback
The relevant information of target object；
Receive the relevant information of the target object of described search server feedback.
16. device according to claim 15, which is characterized in that the data obtaining module is also used to:
The relevant information of the target object of described search server feedback is filtered, it is highest to retain wherein correlation
The information of preset quantity.
17. device according to claim 10, which is characterized in that the superposition processing module is also used to:
According to position of the blinkpunkt of the user eyeball in the AR scene image, by the relevant information of the target object
It is added in the AR scene image at a distance from the blinkpunkt less than in the region of pre-determined distance.
18. device according to claim 17, which is characterized in that the superposition processing module is also used to:
Generate the corresponding pictorial information of relevant information, animation information or the voice messaging of the target object；
By the corresponding pictorial information of the relevant information of the target object, animation information or voice messaging, be added to the use
In the AR scene image within the vision of family.
19. a kind of AR equipment characterized by comprising
Memory, processor, and it is stored in the computer program that can be run on the memory and on the processor,
The processor realizes method as claimed in any one of claims 1-9 wherein when running the computer program.
20. a kind of computer readable storage medium, which is characterized in that it is stored with computer program,
The computer program realizes method as claimed in any one of claims 1-9 wherein when being executed by processor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810974215.1A CN109298780A (en) | 2018-08-24 | 2018-08-24 | Information processing method, device, AR equipment and storage medium based on AR |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810974215.1A CN109298780A (en) | 2018-08-24 | 2018-08-24 | Information processing method, device, AR equipment and storage medium based on AR |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109298780A true CN109298780A (en) | 2019-02-01 |
Family
ID=65165289
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810974215.1A Pending CN109298780A (en) | 2018-08-24 | 2018-08-24 | Information processing method, device, AR equipment and storage medium based on AR |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109298780A (en) |
Cited By (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110286754A (en) * | 2019-06-11 | 2019-09-27 | Oppo广东移动通信有限公司 | Projective techniques and relevant device based on eyeball tracking |
| CN110716645A (en) * | 2019-10-15 | 2020-01-21 | 北京市商汤科技开发有限公司 | Augmented reality data presentation method and device, electronic equipment and storage medium |
| CN110910508A (en) * | 2019-11-20 | 2020-03-24 | 三星电子（中国）研发中心 | Image display method, device and system |
| CN110992859A (en) * | 2019-11-22 | 2020-04-10 | 北京新势界科技有限公司 | Advertising board display method and device based on AR guide |
| CN111475073A (en) * | 2020-05-27 | 2020-07-31 | 众趣（北京）科技有限公司 | Space information preview method and device and electronic equipment |
| CN112069480A (en) * | 2020-08-06 | 2020-12-11 | Oppo广东移动通信有限公司 | Display method and device, storage medium and wearable device |
| CN112184919A (en) * | 2020-10-12 | 2021-01-05 | 中国联合网络通信集团有限公司 | Method, device and storage medium for generating visual field information of AR equipment |
| CN112215965A (en) * | 2020-09-30 | 2021-01-12 | 杭州灵伴科技有限公司 | Scene navigation method, device and computer readable storage medium based on AR |
| CN112991553A (en) * | 2021-03-11 | 2021-06-18 | 深圳市慧鲤科技有限公司 | Information display method and device, electronic equipment and storage medium |
| CN113066193A (en) * | 2021-03-31 | 2021-07-02 | 泰瑞数创科技(北京)有限公司 | Method for enhancing reality on live-action three-dimensional map |
| CN113129112A (en) * | 2021-05-11 | 2021-07-16 | 杭州海康威视数字技术股份有限公司 | Article recommendation method and device and electronic equipment |
| CN113297874A (en) * | 2020-02-21 | 2021-08-24 | 北京七鑫易维信息技术有限公司 | Method and system for mapping information of fixation point |
| CN114115527A (en) * | 2021-10-29 | 2022-03-01 | 北京百度网讯科技有限公司 | Augmented reality AR information display method, device, system and storage medium |
| CN114115532A (en) * | 2021-11-11 | 2022-03-01 | 珊瑚石(上海)视讯科技有限公司 | AR labeling method and system based on display content |
| CN115097903A (en) * | 2022-05-19 | 2022-09-23 | 深圳智华科技发展有限公司 | MR glasses control method and device, MR glasses and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105812778A (en) * | 2015-01-21 | 2016-07-27 | 成都理想境界科技有限公司 | Binocular AR head-mounted display device and information display method therefor |
| CN106774838A (en) * | 2016-11-23 | 2017-05-31 | 上海擎感智能科技有限公司 | The method and device of intelligent glasses and its display information |
| CN107111740A (en) * | 2014-09-29 | 2017-08-29 | 索尼互动娱乐股份有限公司 | For retrieving content item using augmented reality and object recognition and being allowed to the scheme associated with real-world objects |
| CN107272904A (en) * | 2017-06-28 | 2017-10-20 | 联想(北京)有限公司 | A kind of method for displaying image and electronic equipment |
| CN107765842A (en) * | 2016-08-23 | 2018-03-06 | 深圳市掌网科技股份有限公司 | A kind of augmented reality method and system |
-
2018
- 2018-08-24 CN CN201810974215.1A patent/CN109298780A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107111740A (en) * | 2014-09-29 | 2017-08-29 | 索尼互动娱乐股份有限公司 | For retrieving content item using augmented reality and object recognition and being allowed to the scheme associated with real-world objects |
| CN105812778A (en) * | 2015-01-21 | 2016-07-27 | 成都理想境界科技有限公司 | Binocular AR head-mounted display device and information display method therefor |
| CN107765842A (en) * | 2016-08-23 | 2018-03-06 | 深圳市掌网科技股份有限公司 | A kind of augmented reality method and system |
| CN106774838A (en) * | 2016-11-23 | 2017-05-31 | 上海擎感智能科技有限公司 | The method and device of intelligent glasses and its display information |
| CN107272904A (en) * | 2017-06-28 | 2017-10-20 | 联想(北京)有限公司 | A kind of method for displaying image and electronic equipment |
Cited By (21)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110286754A (en) * | 2019-06-11 | 2019-09-27 | Oppo广东移动通信有限公司 | Projective techniques and relevant device based on eyeball tracking |
| CN110716645A (en) * | 2019-10-15 | 2020-01-21 | 北京市商汤科技开发有限公司 | Augmented reality data presentation method and device, electronic equipment and storage medium |
| CN110910508A (en) * | 2019-11-20 | 2020-03-24 | 三星电子（中国）研发中心 | Image display method, device and system |
| CN110910508B (en) * | 2019-11-20 | 2023-04-25 | 三星电子（中国）研发中心 | Image display method, device and system |
| CN110992859A (en) * | 2019-11-22 | 2020-04-10 | 北京新势界科技有限公司 | Advertising board display method and device based on AR guide |
| CN113297874A (en) * | 2020-02-21 | 2021-08-24 | 北京七鑫易维信息技术有限公司 | Method and system for mapping information of fixation point |
| CN111475073A (en) * | 2020-05-27 | 2020-07-31 | 众趣（北京）科技有限公司 | Space information preview method and device and electronic equipment |
| CN112069480A (en) * | 2020-08-06 | 2020-12-11 | Oppo广东移动通信有限公司 | Display method and device, storage medium and wearable device |
| CN112215965A (en) * | 2020-09-30 | 2021-01-12 | 杭州灵伴科技有限公司 | Scene navigation method, device and computer readable storage medium based on AR |
| CN112215965B (en) * | 2020-09-30 | 2024-02-20 | 杭州灵伴科技有限公司 | AR-based scene navigation method, device and computer-readable storage medium |
| CN112184919A (en) * | 2020-10-12 | 2021-01-05 | 中国联合网络通信集团有限公司 | Method, device and storage medium for generating visual field information of AR equipment |
| CN112184919B (en) * | 2020-10-12 | 2023-12-01 | 中国联合网络通信集团有限公司 | Method, device and storage medium for generating visual field information of AR (augmented reality) equipment |
| CN112991553A (en) * | 2021-03-11 | 2021-06-18 | 深圳市慧鲤科技有限公司 | Information display method and device, electronic equipment and storage medium |
| CN113066193A (en) * | 2021-03-31 | 2021-07-02 | 泰瑞数创科技(北京)有限公司 | Method for enhancing reality on live-action three-dimensional map |
| CN113066193B (en) * | 2021-03-31 | 2021-11-05 | 泰瑞数创科技(北京)有限公司 | Method for enhancing reality on live-action three-dimensional map |
| CN113129112A (en) * | 2021-05-11 | 2021-07-16 | 杭州海康威视数字技术股份有限公司 | Article recommendation method and device and electronic equipment |
| CN114115527A (en) * | 2021-10-29 | 2022-03-01 | 北京百度网讯科技有限公司 | Augmented reality AR information display method, device, system and storage medium |
| CN114115532B (en) * | 2021-11-11 | 2023-09-29 | 珊瑚石(上海)视讯科技有限公司 | AR labeling method and system based on display content |
| CN114115532A (en) * | 2021-11-11 | 2022-03-01 | 珊瑚石(上海)视讯科技有限公司 | AR labeling method and system based on display content |
| CN115097903A (en) * | 2022-05-19 | 2022-09-23 | 深圳智华科技发展有限公司 | MR glasses control method and device, MR glasses and storage medium |
| CN115097903B (en) * | 2022-05-19 | 2024-04-05 | 深圳智华科技发展有限公司 | MR glasses control method and device, MR glasses and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109298780A (en) | Information processing method, device, AR equipment and storage medium based on AR | |
| US11776199B2 (en) | Virtual reality environment based manipulation of multi-layered multi-view interactive digital media representations | |
| US10628675B2 (en) | Skeleton detection and tracking via client-server communication | |
| JP6935421B2 (en) | Information display methods, devices, and systems | |
| CN108305317B (en) | Image processing method, device and storage medium | |
| US20180227482A1 (en) | Scene-aware selection of filters and effects for visual digital media content | |
| JP7098604B2 (en) | Automatic tagging of objects in a multi-view interactive digital media representation of a dynamic entity | |
| US9286711B2 (en) | Representing a location at a previous time period using an augmented reality display | |
| US9268406B2 (en) | Virtual spectator experience with a personal audio/visual apparatus | |
| CN105210113B (en) | Monocular vision SLAM with the movement of general and panorama camera | |
| CN115398484A (en) | Cross reality system with geolocation information priority for location | |
| EP4103910A1 (en) | Cross reality system with accurate shared maps | |
| WO2021163289A1 (en) | Cross reality system for large scale environments | |
| US20180047213A1 (en) | Method and apparatus for providing augmented reality-based dynamic service | |
| CN114586071A (en) | Cross-reality system supporting multiple device types | |
| WO2018052665A1 (en) | Virtual reality environment based manipulation of multi-layered multi-view interactive digital media representations | |
| CN115398314A (en) | Cross reality system for map processing using multi-resolution frame descriptors | |
| US20180341808A1 (en) | Visual feature tagging in multi-view interactive digital media representations | |
| KR20160091858A (en) | Reality augmenting method, client device and server | |
| CN112074797A (en) | System and method for anchoring virtual objects to physical locations | |
| CN104978548A (en) | Visual line estimation method and visual line estimation device based on three-dimensional active shape model | |
| CN112037314A (en) | Image display method, image display device, display equipment and computer readable storage medium | |
| CN114332374A (en) | Virtual display method, equipment and storage medium | |
| US20190171281A1 (en) | Image generation program, image generation device, and image generation method | |
| CN111026901A (en) | Learning content searching method and learning equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190201 |
|
| RJ01 | Rejection of invention patent application after publication |