CN114115532B - AR labeling method and system based on display content - Google Patents
AR labeling method and system based on display content Download PDFInfo
- Publication number
- CN114115532B CN114115532B CN202111332392.8A CN202111332392A CN114115532B CN 114115532 B CN114115532 B CN 114115532B CN 202111332392 A CN202111332392 A CN 202111332392A CN 114115532 B CN114115532 B CN 114115532B
- Authority
- CN
- China
- Prior art keywords
- display content
- contents
- content
- trigger set
- display
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/013—Eye tracking input arrangements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
Abstract
The invention discloses an AR labeling method and system based on display content, and belongs to the technical field of intelligent wearable equipment. It comprises the following steps: s1, analyzing display contents to be marked, extracting the marked contents in the display contents, performing AI arrangement, automatically dividing the marked contents into a plurality of trigger sets according to the integrity of the marked contents, and performing background preloaded marking on the set contents by networking. According to the scheme of combining the gazing focus of the user with the active virtual cursor, the gazing focus is triggered slowly, so that unwanted labels are not triggered at will, and then the selected display content can be labeled quickly and accurately by combining secondary determination of the virtual cursor, so that accuracy is guaranteed, and meanwhile, the labeling is performed at the highest speed, and the labeling content is attached to the mind of the user and expected, so that the user experience is good.
Description
Technical Field
The invention relates to an AR labeling method and system based on display content, and belongs to the technical field of intelligent wearable equipment.
Background
Augmented reality (Augmented Reality, abbreviated as AR) is a newer technology content that facilitates the integration between real world information and virtual world information content, which carries out simulated simulation processing on physical information that is otherwise difficult to experience in the spatial range of the real world based on scientific technology such as computers, and the like, and the superposition effectively applies the virtual information content in the real world, and in the process, can be perceived by human senses, thereby realizing a sensory experience that exceeds reality. After overlapping between the real environment and the virtual object, the real environment and the virtual object can exist in the same picture and space simultaneously.
In practical application of AR, superposition of labels on display contents in a display device is the most common way, and the application principle is that the display contents in the current area are subjected to data analysis, the parallel network searches for related contents, the searched related contents are further superimposed on corresponding positions in the display device to form labels, in this process, how to select the desired display contents for labeling is a difficulty, and the existing solution is to use eye tracking technology for interactive selection, however, some defects exist: because people's attention can not concentrate always, lead to the skew that the sight can be irregular, just so can trigger some computer instruction at any time, lead to unwanted mark content to jump out, distraction, on the other hand, current eye moves interactive mode operation inefficiency, adopt single eye blink or the action of both eyes blink to click and operate can easily cause tired of eyes, and blink also belongs to natural reaction, and when eyes receive the stimulus, blink frequency is more uncontrolled, thereby cause the maloperation easily, just so make the user can not carry out the mark of display content according to own will, influence user experience.
Disclosure of Invention
The invention aims at the technical problems, so that an AR labeling method and an AR labeling system based on display content are provided.
The technical problems to be solved by the invention are realized by adopting the following technical scheme:
an AR labeling method based on display content comprises the following steps:
s1, analyzing display contents to be marked, extracting the marked contents in the display contents, performing AI arrangement, automatically dividing the marked contents into a plurality of trigger sets according to the integrity of the marked contents, and performing background preloaded marking on the set contents by networking;
s2, capturing a user' S gazing focus by an eyeball tracking module of the device, wherein after the time that the gazing focus is concentrated on a trigger set exceeds a set time, the display content mapped by the trigger set is highlighted and used for indicating that the display content at the place has a callable content, and after the highlighting lasts for the set time, the highlighting effect is hidden;
s3, capturing finger gesture information by an image shooting module of the device, forming an invisible virtual cursor at the top end of a fingertip of the finger, enabling a mark on the corresponding trigger set to enter a pre-loading state when the virtual cursor stays on the trigger set, loading and displaying the mark content after the trigger set stays for a set time, and hiding the mark content after the virtual cursor leaves the trigger set for a set time;
s4, when the gazing focus and the virtual cursor captured by the eyeball tracking module are concentrated on one trigger set, marking content is displayed in a display content nearby area mapped by the trigger set immediately;
and S5, repeating the step S2 or S3 when one of the gazing focus and the virtual cursor leaves the trigger set, and restoring the display content to be the same when the gazing focus and the virtual cursor leave the trigger set.
As a preferred example, the noted content includes graphics, animations, video, text, and icons.
As a preferable example, in S2, the setting time of focusing the gaze on the trigger set is 3 to 5S, and the setting time of highlighting continuation is 1 to 3S.
As a preferred example, in the S2, after the highlight effect is removed, the trigger set receives the highlight wake of the gazing focus again after 10-15S.
As a preferred example, in S2, the highlighted content is quickly converted into the labeling content display by the remote operation device.
As a preferable example, in S3, the setting time for the virtual cursor to stay on the trigger set is 5-8S, and the setting time for the virtual cursor to leave the trigger set is 1-3S.
As a preferred example, in S3, the captured finger posture information does not include the posture information of the thumb.
An AR labeling system based on display content comprises a display module, a system processing module, a signal module, an eyeball tracking module and an image shooting module;
the display module comprises a transparent screen and is used for superposing AR labeling content on display content in the real world;
the system processing module comprises a logic and a relative computer memory of control instructions, and is used for receiving and processing signals from the sensor, providing display signals for the display module and making AR labeling content;
the signal module comprises at least two of a 5G network communication module, a wireless module, a Bluetooth module or an infrared module and is used for being connected with external signals;
the eyeball tracking module is used for tracking the gazing focuses of the eyes of the user in real time and converting the gazing focuses into coordinate signals to be transmitted to the system processing module;
the image shooting module is used for extracting display content and capturing finger gesture information, the extracted display content is converted into a processable signal and transmitted to the system processing module, and the captured finger gesture information is converted into a coordinate signal and transmitted to the system processing module.
As a preferred example, a remote operation device is also included, which includes a smart finger ring or a smart bracelet/watch, and is connected to the signal module by a wireless signal.
The beneficial effects of the invention are as follows: according to the scheme of combining the gazing focus of the user with the active virtual cursor, the gazing focus is triggered slowly, so that unwanted labels are not triggered at will, and then the selected display content can be labeled quickly and accurately by combining secondary determination of the virtual cursor, so that accuracy is guaranteed, and meanwhile, the labeling is performed at the highest speed, and the labeling content is attached to the mind of the user and expected, so that the user experience is good.
Drawings
FIG. 1 is a workflow diagram of the present invention;
fig. 2 is a schematic structural view of the present invention.
Detailed Description
The invention will be further described with reference to the following detailed drawings, in order to make the technical means, the creation characteristics, the achievement of the purpose and the effect of the invention easy to understand.
The invention is implemented on the basis of AR glasses or smart phones/tablets, which must be provided with associated image capturing means for capturing the display content and for capturing the eye movements of the user.
As shown in fig. 1, an AR labeling method based on display content includes the following steps:
s1, analyzing display contents to be marked, extracting the marked contents in the display contents, performing AI arrangement, automatically dividing according to the integrity of the marked contents, such as a word, a sentence, a single graph and other units with complete meanings, editing the units into a plurality of trigger sets, performing background preloading marking on the contents of the trigger sets by a parallel network, caching the marked contents in a computer memory, converting the whole display contents into two-dimensional plane data, converting the plurality of trigger sets into coordinate sets according to original corresponding positions on the display contents, and embedding the coordinate sets in the positions corresponding to the two-dimensional planes;
s2, capturing a gazing focus of a user by an eyeball tracking module of the equipment, converting a gazing focus position signal into a coordinate signal on a two-dimensional plane so as to judge whether the gazing focus coincides with a trigger set, and when the time of the gazing focus on the trigger set exceeds a set time, highlighting display contents mapped by the trigger set, wherein the display contents are used for indicating that the display contents can be marked, after the highlighting lasts for the set time, the highlighting effect is hidden, and when the time of the gazing focus on the trigger set is lower than the set time, the display contents do not react, so that misoperation caused by jumping of a sight is prevented;
s3, capturing finger gesture information by an image capturing module of the device, capturing and tracking the peripheral outline of the finger through an image recognition technology, determining a plurality of measuring points on the peripheral outline based on a template matching algorithm and based on an artificial neural network, determining coordinates on a two-dimensional plane by the plurality of measuring points at the moment, recognizing the measuring points at the top end of the finger fingertip as invisible virtual cursors, enabling labels on the corresponding trigger set to enter a pre-loading state when the virtual cursors stay on the trigger set, loading and displaying the label content after the trigger set stays for a set time, and hiding the label content after the virtual cursors leave the trigger set for a set time;
s4, when the gazing focus and the virtual cursor captured by the eyeball tracking module are concentrated on one trigger set, namely, under the condition of no consideration of depth, the two-dimensional coordinates of the gazing focus and the two-dimensional coordinates of the virtual cursor are both positioned in the coordinate set of the trigger set, and at the moment, marking contents are immediately displayed in the area near the display contents mapped by the trigger set;
and S5, repeating the step S2 or S3 when one of the gazing focus and the virtual cursor leaves the trigger set, and restoring the display content to be the same when the gazing focus and the virtual cursor leave the trigger set.
The annotated content includes graphics, animations, video, text and icons.
And S2, setting time of focusing the gazing focus on the trigger set is 3-5S, setting time of highlighting display duration is 1-3S, and after the highlighting effect is hidden, the trigger set receives highlight wakeup of the gazing focus again after 10-15S.
By adopting the scheme, when the reading or watching speed of the user is comprehensively considered and the focusing focus is concentrated at about 4s, the influence of short sight line deviation is avoided, the corresponding labeling time is not delayed too much, and meanwhile, the setting of the dormancy time after the highlight effect is hidden is to prevent the user from repeatedly carrying out highlight to disperse the attention of the user when the user focuses on watching a section of display content.
In S2, the highlighted content is quickly converted into the marked content to be displayed through the remote operation equipment, and the specific principle is that the remote operation equipment performs active marking operation through triggering or gestures at the moment under the highlight state of the displayed content, so that the content marking can be conveniently and quickly completed.
In S3, the setting time of the virtual cursor staying on the trigger set is 5-8S, the setting time of the virtual cursor leaving the trigger set is 1-3S, the setting time of the triggering time of the virtual cursor is slightly longer than the triggering time of the fixation focus, the finger gesture is prevented from being accidentally shot by the image shooting module and immediately triggering the mark to influence the sight line, and meanwhile, the effect of marking the target display content can be achieved after a period of time.
In S3, the captured finger gesture information does not comprise the gesture information of the thumb, because in a normal use scene, a user does not need to use the thumb when reading and understanding with the assistance of fingers, and generally only uses the index finger and the middle finger, and the ring finger and the little finger are contracted, the method can shield the gesture information capture of the thumb, and prevent the gesture information of the thumb from mistakenly touching irrelevant display content labels.
As shown in fig. 2, an AR labeling system based on display content includes a display module, a system processing module, a signal module, an eye tracking module, and an image capturing module;
the display module comprises a transparent screen for superimposing AR labeling content on display content in the real world;
the system processing module comprises a logic and control instruction related computer memory, is used for receiving and processing signals from the sensor, providing display signals for the display module and making AR labeling content;
the signal module comprises at least two of a 5G network communication module, a wireless module, a Bluetooth module or an infrared module and is used for being connected with external signals, including networking inquiry marking data, data exchange with an information terminal, information instruction of a remote operation device receiving and the like;
the eyeball tracking module is used for tracking the gazing focuses of the eyes of a user in real time, converting the gazing focuses into coordinate signals and transmitting the coordinate signals to the system processing module, and the main equipment comprises infrared equipment and image acquisition equipment;
the image shooting module is used for extracting display content and capturing finger gesture information, the extracted display content is converted into a processable signal and transmitted to the system processing module, and the captured finger gesture information is converted into a coordinate signal and transmitted to the system processing module.
The remote operation device is connected with the signal module through a wireless signal, the remote operation device is to be used for a wearable device such as an intelligent finger ring or an intelligent hand ring/watch, remote instruction operation is carried out through pressing and clicking or gesture movement modes of the device, and the remote instruction operation is carried out through direct marking of highlighted contents.
The foregoing has shown and described the basic principles and main features of the present invention and the advantages of the present invention. It will be appreciated by persons skilled in the art that the present invention is not limited to the embodiments described above, but is capable of numerous variations and modifications without departing from the spirit and scope of the invention as hereinafter claimed. The scope of the invention is defined by the appended claims and equivalents thereof.
Claims (7)
1. An AR labeling method based on display content is characterized by comprising the following steps: the method comprises the following steps:
s1, analyzing display contents to be marked, extracting the marked contents in the display contents, performing AI arrangement, automatically dividing the marked contents into a plurality of trigger sets according to the integrity of the marked contents, and performing background preloaded marking on the set contents by networking;
s2, capturing a user' S gazing focus by an eyeball tracking module of the device, wherein after the time that the gazing focus is concentrated on a trigger set exceeds a set time, the display content mapped by the trigger set is highlighted and used for indicating that the display content at the place has a callable content, and after the highlighting lasts for the set time, the highlighting effect is hidden;
s3, capturing finger gesture information by an image shooting module of the device, forming an invisible virtual cursor at the top end of a fingertip of the finger, enabling a mark on the corresponding trigger set to enter a pre-loading state when the virtual cursor stays on the trigger set, loading and displaying the mark content after the trigger set stays for a set time, and hiding the mark content after the virtual cursor leaves the trigger set for a set time;
s4, when the gazing focus and the virtual cursor captured by the eyeball tracking module are concentrated on one trigger set, marking content is displayed in a display content nearby area mapped by the trigger set immediately;
and S5, repeating the step S2 or S3 when one of the gazing focus and the virtual cursor leaves the trigger set, and restoring the display content to be the same when the gazing focus and the virtual cursor leave the trigger set.
2. The AR labeling method based on display content according to claim 1, wherein: the noted content includes graphics, animations, video, text, and icons.
3. The AR labeling method based on display content according to claim 1, wherein: in S2, the setting time of focusing on the trigger set is 3-5S, and the setting time of highlighting duration is 1-3S.
4. The AR labeling method based on display content according to claim 1, wherein: in the step S2, after the highlight effect is hidden, the trigger set receives highlight wake-up of the gazing focus again after 10-15S.
5. The AR labeling method based on display content according to claim 1, wherein: in the step S2, the highlighted content is quickly converted into marked content to be displayed through the remote operation device.
6. The AR labeling method based on display content according to claim 1, wherein: in the step S3, the setting time of the virtual cursor staying on the trigger set is 5-8S, and the setting time of the virtual cursor leaving the trigger set is 1-3S.
7. The AR labeling method based on display content according to claim 1, wherein: in the step S3, the captured finger gesture information does not include gesture information of the thumb.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111332392.8A CN114115532B (en) | 2021-11-11 | 2021-11-11 | AR labeling method and system based on display content |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111332392.8A CN114115532B (en) | 2021-11-11 | 2021-11-11 | AR labeling method and system based on display content |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114115532A CN114115532A (en) | 2022-03-01 |
| CN114115532B true CN114115532B (en) | 2023-09-29 |
Family
ID=80378242
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111332392.8A Active CN114115532B (en) | 2021-11-11 | 2021-11-11 | AR labeling method and system based on display content |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114115532B (en) |
Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2016064073A1 (en) * | 2014-10-22 | 2016-04-28 | 윤영기 | Smart glasses on which display and camera are mounted, and a space touch inputting and correction method using same |
| CN106814854A (en) * | 2016-12-29 | 2017-06-09 | 杭州联络互动信息科技股份有限公司 | A kind of method and device for preventing maloperation |
| US10061352B1 (en) * | 2017-08-14 | 2018-08-28 | Oculus Vr, Llc | Distributed augmented reality system |
| CN108829239A (en) * | 2018-05-07 | 2018-11-16 | 北京七鑫易维信息技术有限公司 | Control method, device and the terminal of terminal |
| CN109298780A (en) * | 2018-08-24 | 2019-02-01 | 百度在线网络技术（北京）有限公司 | Information processing method, device, AR equipment and storage medium based on AR |
| CN110187855A (en) * | 2019-05-28 | 2019-08-30 | 武汉市天蝎科技有限公司 | The intelligent adjusting method for avoiding hologram block vision of near-eye display device |
| KR20190128962A (en) * | 2018-05-09 | 2019-11-19 | 서강대학교산학협력단 | METHOD AND WEARABLE DISPLAY APPARATUS FOR PROVIDING eBOOK BASED ON AUGMENTED REALLITY |
| CN111931579A (en) * | 2020-07-09 | 2020-11-13 | 上海交通大学 | Automatic driving assistance system and method using eye tracking and gesture recognition technology |
| CN111949131A (en) * | 2020-08-17 | 2020-11-17 | 陈涛 | Eye movement interaction method, system and equipment based on eye movement tracking technology |
| WO2021073743A1 (en) * | 2019-10-17 | 2021-04-22 | Huawei Technologies Co., Ltd. | Determining user input based on hand gestures and eye tracking |
| CN112817447A (en) * | 2021-01-25 | 2021-05-18 | 暗物智能科技(广州)有限公司 | AR content display method and system |
| KR20210073429A (en) * | 2019-12-10 | 2021-06-18 | 한국전자기술연구원 | Integration Interface Method and System based on Eye tracking and Gesture recognition for Wearable Augmented Reality Device |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9323325B2 (en) * | 2011-08-30 | 2016-04-26 | Microsoft Technology Licensing, Llc | Enhancing an object of interest in a see-through, mixed reality display device |
| US9201500B2 (en) * | 2012-09-28 | 2015-12-01 | Intel Corporation | Multi-modal touch screen emulator |
| KR101519290B1 (en) * | 2014-04-09 | 2015-05-11 | 현대자동차주식회사 | Method for Controlling HUD for Vehicle |
-
2021
- 2021-11-11 CN CN202111332392.8A patent/CN114115532B/en active Active
Patent Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2016064073A1 (en) * | 2014-10-22 | 2016-04-28 | 윤영기 | Smart glasses on which display and camera are mounted, and a space touch inputting and correction method using same |
| CN106814854A (en) * | 2016-12-29 | 2017-06-09 | 杭州联络互动信息科技股份有限公司 | A kind of method and device for preventing maloperation |
| US10061352B1 (en) * | 2017-08-14 | 2018-08-28 | Oculus Vr, Llc | Distributed augmented reality system |
| CN108829239A (en) * | 2018-05-07 | 2018-11-16 | 北京七鑫易维信息技术有限公司 | Control method, device and the terminal of terminal |
| KR20190128962A (en) * | 2018-05-09 | 2019-11-19 | 서강대학교산학협력단 | METHOD AND WEARABLE DISPLAY APPARATUS FOR PROVIDING eBOOK BASED ON AUGMENTED REALLITY |
| CN109298780A (en) * | 2018-08-24 | 2019-02-01 | 百度在线网络技术（北京）有限公司 | Information processing method, device, AR equipment and storage medium based on AR |
| CN110187855A (en) * | 2019-05-28 | 2019-08-30 | 武汉市天蝎科技有限公司 | The intelligent adjusting method for avoiding hologram block vision of near-eye display device |
| WO2021073743A1 (en) * | 2019-10-17 | 2021-04-22 | Huawei Technologies Co., Ltd. | Determining user input based on hand gestures and eye tracking |
| KR20210073429A (en) * | 2019-12-10 | 2021-06-18 | 한국전자기술연구원 | Integration Interface Method and System based on Eye tracking and Gesture recognition for Wearable Augmented Reality Device |
| CN111931579A (en) * | 2020-07-09 | 2020-11-13 | 上海交通大学 | Automatic driving assistance system and method using eye tracking and gesture recognition technology |
| CN111949131A (en) * | 2020-08-17 | 2020-11-17 | 陈涛 | Eye movement interaction method, system and equipment based on eye movement tracking technology |
| CN112817447A (en) * | 2021-01-25 | 2021-05-18 | 暗物智能科技(广州)有限公司 | AR content display method and system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114115532A (en) | 2022-03-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Yang et al. | Gesture interaction in virtual reality | |
| US9671872B2 (en) | Gesture recognition method, gesture recognition system, terminal device and wearable device | |
| CN107479691B (en) | Interaction method, intelligent glasses and storage device thereof | |
| US9857971B2 (en) | System and method for receiving user input and program storage medium thereof | |
| CN109375765B (en) | Eyeball tracking interaction method and device | |
| CN107562186B (en) | 3D campus navigation method for emotion operation based on attention identification | |
| CN108616712B (en) | Camera-based interface operation method, device, equipment and storage medium | |
| CN102915111A (en) | Wrist gesture control system and method | |
| US20240077948A1 (en) | Gesture-based display interface control method and apparatus, device and storage medium | |
| CN108027655A (en) | Information processing system, information processing equipment, control method and program | |
| CN107450717B (en) | Information processing method and wearable device | |
| Arai et al. | Eye-based human computer interaction allowing phoning, reading e-book/e-comic/e-learning, internet browsing, and tv information extraction | |
| CN114821753B (en) | Eye movement interaction system based on visual image information | |
| CN108829239A (en) | Control method, device and the terminal of terminal | |
| CN106681509A (en) | Interface operating method and system | |
| KR101488662B1 (en) | Device and method for providing interface interacting with a user using natural user interface device | |
| CN111901518B (en) | Display method and device and electronic equipment | |
| CN114115532B (en) | AR labeling method and system based on display content | |
| US11328187B2 (en) | Information processing apparatus and information processing method | |
| Raees et al. | THE-3DI: Tracing head and eyes for 3D interactions: An interaction technique for virtual environments | |
| Annachhatre et al. | Virtual Mouse Using Hand Gesture Recognition-A Systematic Literature Review | |
| CN115047966A (en) | Interaction method, electronic equipment and interaction system | |
| Strumiłło et al. | A vision-based head movement tracking system for human-computer interfacing | |
| CN204270276U (en) | A kind of human-computer interaction device based on radio-frequency (RF) identification | |
| CN111290579B (en) | Control method and device of virtual content, electronic equipment and computer readable medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |