JP2021121923A - Augmented reality system and method for tracking biometric data - Google Patents

Abstract

To provide an augmented reality (AR) display system and a method for utilizing biometric data to facilitate business transactions conducted through an AR device.SOLUTION: A method of conducting a transaction through an augmented reality display device includes: capturing biometric data from a user; determining, based on at least partially the captured biometric data, an identity of the user; authenticating the user for the transaction based on the determined identity; and transmitting a set of data regarding the transaction to a financial institution. The biometric data is an iris pattern.SELECTED DRAWING: Figure 7

Description

The present disclosure relates to systems and methods for using biometric data to facilitate commerce conducted through augmented reality (AR) devices.

Modern computing and display technology presents digitally reproduced images or parts thereof to users in a manner that appears to be real, or in a manner that can be perceived as such, the so-called "virtual reality". Promotes the development of systems for "augmented reality" experiences. Virtual reality, or "VR" scenarios, typically involves the presentation of digital or virtual image information without transparency to other real-world visual inputs. Augmented reality, or "AR" scenarios, typically involve the presentation of digital or virtual image information as an extension of the visualization of the real world around the user.

For example, referring to FIG. 1, an augmented reality scene is depicted, and users of AR technology can
See a real-world park-like setting featuring a concrete platform 1120 against the backdrop of people, trees and buildings. In addition to these items, users of AR technology also perceive the robot image 1110 standing on the real-world platform 1120 and the flying cartoon-like avatar character 2, but these elements (2, 1110) are , Does not exist in the real world. The human visual perception system is very complex, such augmented reality that promotes the comfortable, natural-feeling, and rich presentation of virtual image elements among other virtual or real-world image elements. Generating a scene is difficult.

It is envisioned that such an AR device could be used to present any type of virtual content to the user. In one or more embodiments, the AR device may be used in the context of various gaming applications, allowing the user to participate in a single player or multiplayer video / augmented reality game that mimics a real situation. .. For example, rather than playing a video game on a personal computer, an AR user can play the game on a larger scale in conditions that are very similar to the real world (
For example, a "real size" 3D monster can appear from behind an actual building when an AR user is walking in the park). In fact, this significantly improves the credibility and enjoyment of the gaming experience.

FIG. 1 illustrates the potential of AR devices in the context of gaming applications,
AR devices can be used in many other applications and can be expected to replace everyday computing devices (eg, personal computers, mobile phones, tablet devices, etc.). By strategically placing virtual content in the user's field of view
AR devices allow users to simultaneously connect to their physical environment while performing various computing tasks (eg, checking email, looking up terms on the web, etc.).
It can be considered a walking personal computer that allows you to have conference calls with other AR users, watch movies, etc.). For example, rather than being constrained by physical devices on the desk, AR users are "on the move" (eg, while walking, on a daily commute, away from the computer in a physical location other than the office, etc. ), And it is still possible to have a video conference with a friend by pulling out the virtual email screen and, for example, checking the email or virtually presetting the screen on the AR device. , Or in another example, it is possible to build a virtual office in a makeshift location. Many similar virtual reality / augmented reality scenarios can be envisioned.

This shift in the nature of lower-tier computing technology has its own advantages and challenges. To present augmented reality scenes such as those mentioned above, taking into account the physiological limitations of the human visual system, AR devices associate desired virtual content with one or more real objects within the user's physical environment. You must be aware of the user's physical surroundings in order to project. To this end, AR devices typically include a variety of tracking devices (eg, for example.
Equipped with eye tracking devices, GPS, etc.), cameras (eg, field cameras, infrared cameras, depth cameras, etc.), and sensors (eg, accelerometers, gyroscopes, etc.) and associated with various real objects within the user's surroundings. It assesses the user's position, orientation, distance, etc., and detects and identifies real-world objects and other such functionality.

If the AR device is configured to track various types of data about the AR user and its surroundings, in one or more embodiments, this data advantageously requires minimal input from the user. Ensure that it is done, and with little or no interruption in the user's AR
It can be utilized to assist various types of transactions of the user without causing it to occur in the experience.

More specifically, traditional transactions (financial or other) typically involve the user in some form of monetary token (eg, cash, check, credit card, etc.), in some cases, in order to participate in a commercial transaction. Require physical carrying of identification (eg, driver's license, etc.), and authentication (eg, signature, etc.). Suppose a user goes inside a department store. To make any kind of purchase, the user typically picks up the item, puts the item in the cart, walks to the cashier, lines up in front of the cashier, and has some cashiers. Wait for the item to be scanned, remove the credit card, provide identification, sign the credit card receipt, and keep the receipt for future returns of the item. In traditional financial transactions, these steps are necessary, but time consuming and inefficient, and in some cases these steps deter or prohibit users from making purchases (eg,). The user does not carry a financial token or ID card, etc.). However, in the context of AR devices, these steps are redundant and unnecessary. In one or more embodiments, AR
The device may be configured to allow the user to seamlessly carry out many types of transactions without requiring the user to perform the tedious steps described above.

Therefore, there is a need for better solutions to help AR users participate in routine transactions.

Embodiments of the invention cover devices, systems, and methods for facilitating virtual reality and / or augmented reality interactions for one or more users.

On the one hand, the method of conducting a transaction through an augmented reality device is to capture biometric data from the user and to determine the user's identity based at least in part on the captured biometric data. Includes authenticating the user for a transaction based on the identification made.

In one or more embodiments, the method further comprises transmitting a set of data about the transaction to a financial institution. In one or more embodiments, the biometric data is an iris pattern. In one or more embodiments, the biometric data is a voice recording of the user. 1
In one or more embodiments, the biometric data is a retinal signature. In one or more embodiments, the biometric data is a property associated with the user's skin.

In one or more embodiments, the biometric data captures the movement of the user's eyes.
Captured through one or more eye tracking cameras. In one or more embodiments, the biometric data is a pattern of movement of the user's eyes. In one or more embodiments, the biometric data is a blink pattern of the user's eyes.

In one or more embodiments, the augmented reality device is head-mounted and the augmented reality device is individually calibrated for the user. In one or more embodiments, the biometric data is compared to predetermined data about the user. In one or more embodiments, the predetermined data is a known signature transfer in the user's eye.

In one or more embodiments, the predetermined data is a known iris pattern. In one or more embodiments, the predetermined data is a known retinal pattern. In one or more embodiments, the method further detects the user's desire to make a transaction, requests biometric data from the user at least in part based on the detected desire, and biometrically. Comparing metric data with predetermined biometric data to produce a result, including that the user is authenticated based on the result, at least in part.

In one or more embodiments, the transaction is a commercial transaction. In one or more embodiments, the method is further to communicate the user's authentication to the financial institution associated with the user, who pays on behalf of the user, at least in part, based on the authentication. Including doing.
In one or more embodiments, the financial institution transmits the payment to one or more distributors indicated by the user.

In one or more embodiments, the method further comprises detecting an interruption event or transaction event associated with an augmented reality device. In one or more embodiments, the method further comprises capturing new biometric data from the user in order to reauthenticate the user, at least in part, based on the detected event. In one or more embodiments,
Activity interruptions are detected, at least in part, based on the removal of the augmented reality device from the user's head.

In one or more embodiments, the interruption of activity is detected, at least in part, based on the loss of connectivity of the augmented reality device with the network. In one or more embodiments, the transaction event is detected, at least in part, based on the explicit approval of the transaction by the user. In one or more embodiments, the transaction event is detected, at least in part, based on the heatmap associated with the user's gaze.

In one or more embodiments, the transaction event is detected, at least in part, based on user input received through the augmented reality device. In one or more embodiments, the user input comprises an eye gesture. In one or more embodiments, the user input comprises a hand gesture.

In another aspect, the augmented reality display system is operably coupled to a biometric data tracking device for capturing biometric data from the user and the biometric data tracking device to process the captured biometric data. It includes a processor for determining user identification, at least partially based on captured biometric data, and at least a server for communicating with financial institutions and authenticating users for transactions.

In one or more embodiments, the biometric data is eye movement data. In one or more embodiments, the biometric data corresponds to an image of the user's iris. In one or more embodiments, the server also transmits a set of data about the transaction to the financial institution. In one or more embodiments, the biometric data is an iris pattern.

In one or more embodiments, the biometric data is a voice recording of the user. In one or more embodiments, the biometric data is a retinal signature. In one or more embodiments, the biometric data is a property associated with the user's skin. In one or more embodiments, the biometric tracking device comprises one or more eye tracking cameras to capture the movement of the user's eyes. In one or more embodiments, the biometric data is a pattern of movement of the user's eyes.

In one or more embodiments, the biometric data is a blink pattern of the user's eyes. In one or more embodiments, the augmented reality display system is head-mounted and the augmented reality display system is individually calibrated for the user. In one or more embodiments, the processor also compares biometric data with predetermined data about the user. In one or more embodiments, the predetermined data is a known signature transfer in the user's eye.
In one or more embodiments, the predetermined data is a known iris pattern. In one or more embodiments, the predetermined data is a known retinal pattern. In one or more embodiments, the processor detects that the user wants to make a transaction, and the processor requests biometric data from the user, at least in part, based on the detection. Further, the processor compares the biometric data with the predetermined biometric data and authenticates the user based on the comparison, at least in part.

In one or more embodiments, the transaction is a commercial transaction. In one or more embodiments, the processor communicates the user's authentication to the financial institution associated with the user, who makes payments on behalf of the user, at least in part based on the authentication. In one or more embodiments, the financial institution transmits the payment to one or more distributors indicated by the user.

In one or more embodiments, the processor detects interruption or trading events associated with the augmented reality device, and the biometric tracking device reauthenticates the user based at least in part on the detected events. To capture new biometric data from users. In one or more embodiments, the interruption of activity is detected, at least in part, based on the removal of the augmented reality device from the user's head.

In one or more embodiments, the interruption of activity is detected, at least in part, based on the loss of connectivity of the augmented reality device with the network. In one or more embodiments, the transaction event is detected, at least in part, based on the explicit approval of the transaction by the user. In one or more embodiments, the transaction event is detected, at least in part, based on the heatmap associated with the user's gaze. In one or more embodiments, the transaction event is detected, at least in part, based on user input received through the augmented reality device. In one or more embodiments, the user input comprises an eye gesture. In one or more embodiments, the user input comprises a hand gesture.

In one or more embodiments, the biometric tracking device comprises an eye tracking system. In one or more embodiments, the biometric tracking device comprises a tactile device. In one or more embodiments, the biometric tracking device comprises a sensor that measures physiological data about the user's eye.
The present specification also provides, for example, the following items.
(Item 1)
A method of conducting a transaction through an augmented reality device, said method.
Capturing biometric data from users and
Determining the identification of the user based at least in part on the captured biometric data.
A method comprising authenticating the user for the transaction based on the determined identification.
(Item 2)
The method of item 1, further comprising transmitting a set of data relating to the transaction to a financial institution.
(Item 3)
The method according to item 1, wherein the biometric data is an iris pattern.
(Item 4)
The method according to item 1, wherein the biometric data is a voice recording of the user.
(Item 5)
The method according to item 1, wherein the biometric data is a retinal signature.
(Item 6)
The method of item 1, wherein the biometric data is a property associated with the user's skin.
(Item 7)
The method of item 1, wherein the biometric data is captured through one or more eye tracking cameras that capture the movement of the user's eyes.
(Item 8)
The method according to item 1, wherein the biometric data is a pattern of eye movement of the user.
(Item 9)
The method according to item 1, wherein the biometric data is a blink pattern of the user's eyes.
(Item 10)
The method of item 1, wherein the augmented reality device is head-mounted and the augmented reality device is individually calibrated for the user.
(Item 11)
The method of item 1, wherein the biometric data is compared with predetermined data about the user.
(Item 12)
The method of item 11, wherein the predetermined data is a known signature movement of the user's eye.
(Item 13)
The method of item 11, wherein the predetermined data is a known iris pattern.
(Item 14)
The method of item 11, wherein the predetermined data is a known retinal pattern.
(Item 15)
Detecting the user's desire to make a transaction
Requesting the biometric data from the user, at least in part, based on the detected desire.
The item 1 further comprises comparing the biometric data with predetermined biometric data and generating a result, wherein the user is authenticated based on the result at least in part. the method of.
(Item 16)
The method according to item 1, wherein the transaction is a commercial transaction.
(Item 17)
The first item, wherein the user's authentication further comprises communicating the user's authentication to a financial institution associated with the user, the financial institution making payments on behalf of the user, at least in part based on the authentication. Method.
(Item 18)
17. The method of item 17, wherein the financial institution transmits the payment to one or more sellers indicated by the user.
(Item 19)
To detect interruption or transaction events associated with the augmented reality device,
The method of item 1, further comprising capturing new biometric data from the user in order to reauthenticate the user, at least in part, based on the detected event.
(Item 20)
19. The method of item 19, wherein the interruption of the activity is detected, at least in part, based on the removal of the augmented reality device from the user's head.
(Item 21)
19. The method of item 19, wherein the interruption in activity is detected, at least in part, based on the loss of connectivity of the augmented reality device with the network.
(Item 22)
19. The method of item 19, wherein the transaction event is detected, at least in part, based on the explicit approval of the transaction by the user.
(Item 23)
19. The method of item 19, wherein the transaction event is detected, at least in part, based on a heatmap associated with the user's gaze.
(Item 24)
19. The method of item 19, wherein the transaction event is detected, at least in part, based on user input received through the augmented reality device.
(Item 25)
24. The method of item 24, wherein the user input comprises an eye gesture.
(Item 26)
24. The method of item 24, wherein the user input comprises a hand gesture.
(Item 27)
The method of item 1-26, which is implemented as a system having means for implementing the method steps.
(Item 28)
The method of item 1-26, which is implemented as a computer program product comprising a computer-enabled storage medium having executable code for performing the method steps.

Additional and other purposes, features, and advantages of the invention are the embodiments for carrying out the invention.
Explained in the drawings and claims.

The drawings illustrate the design and availability of various embodiments of the invention. Note that figures are not necessarily drawn to exact scale and elements of similar structure or function are represented by similar reference numbers throughout the figure. In order to gain a deeper understanding of the aforementioned and other advantages of the various embodiments of the present invention as well as the method of obtaining the object, embodiments for carrying out the aforementioned invention in a simple manner are illustrated in the accompanying drawings. It will be given by reference to the embodiments. With the understanding that these drawings depict only typical embodiments of the invention and are therefore not considered to be a limitation of its scope, the invention provides additional specificity and detail through the use of accompanying drawings. Will be described and explained with.
FIG. 1 illustrates an exemplary augmented reality scene displayed to the user. FIG. 2A-2D illustrates various configurations of an exemplary augmented reality device. FIG. 2A-2D illustrates various configurations of an exemplary augmented reality device. FIG. 2A-2D illustrates various configurations of an exemplary augmented reality device. FIG. 2A-2D illustrates various configurations of an exemplary augmented reality device. FIG. 3 illustrates an augmented reality device that communicates with one or more servers in the cloud, according to one embodiment. 4A-4D illustrate various eye and head measurements made to configure an augmented reality device for a particular user. FIG. 5 shows a plan view of various components of an augmented reality device according to one embodiment. FIG. 6 shows the system architecture of an augmented reality system for conducting commercial transactions according to one embodiment. FIG. 7 is an exemplary flowchart illustrating how to conduct a commercial transaction through an augmented reality device. 8A and 8B illustrate an exemplary eye identification method for identifying a user according to one embodiment. 8A and 8B illustrate an exemplary eye identification method for identifying a user according to one embodiment. FIG. 9 illustrates an exemplary flow chart illustrating a method of authenticating a user using eye movement according to one embodiment. 10A-10I illustrate a series of process flow diagrams that depict an exemplary scenario of conducting a commercial transaction using an augmented reality device. 10A-10I illustrate a series of process flow diagrams that depict an exemplary scenario of conducting a commercial transaction using an augmented reality device. 10A-10I illustrate a series of process flow diagrams that depict an exemplary scenario of conducting a commercial transaction using an augmented reality device. 10A-10I illustrate a series of process flow diagrams that depict an exemplary scenario of conducting a commercial transaction using an augmented reality device. 10A-10I illustrate a series of process flow diagrams that depict an exemplary scenario of conducting a commercial transaction using an augmented reality device. 10A-10I illustrate a series of process flow diagrams that depict an exemplary scenario of conducting a commercial transaction using an augmented reality device. 10A-10I illustrate a series of process flow diagrams that depict an exemplary scenario of conducting a commercial transaction using an augmented reality device. 10A-10I illustrate a series of process flow diagrams that depict an exemplary scenario of conducting a commercial transaction using an augmented reality device. 10A-10I illustrate a series of process flow diagrams that depict an exemplary scenario of conducting a commercial transaction using an augmented reality device.

Various embodiments of the present invention cover methods, systems, and manufactured products that implement a multi-scenario physical cognitive design of electronic circuit design in a single embodiment or multiple embodiments. Other objects, features, and advantages of the invention are described in embodiments, drawings, and claims for carrying out the invention.

Here, various embodiments will be described in detail with reference to the drawings provided as exemplary embodiments of the invention to allow one of ordinary skill in the art to practice the invention. .. It should be noted that the figures and examples below do not imply limiting the scope of the invention. Some elements of the invention are known components (or methods or processes).
Only those parts of such known components (or methods or processes) necessary for the understanding of the present invention are described and such known components if they can be partially or fully implemented using. A detailed description of other parts (or methods or processes) will be omitted so as not to obscure the invention. In addition, various embodiments include current and future known equivalents of the components referenced herein by way of example.

Disclosed are methods and systems that track biometric data associated with AR users and utilize the biometric data to assist in commerce. In one or more embodiments, the AR device utilizes eye identification techniques (eg, iris pattern, binocular detachment movement, eye movement, pyramidal and rod cell patterns, eye movement patterns, etc.). Can authenticate the user for purchase. Advantageously, this type of user authentication minimizes the frictional cost of conducting a commercial transaction and allows the user to make a seamless purchase with minimal effort and / or interruption (eg). , Physical store, online store,
In response to advertisements etc.). Although the following disclosure focuses primarily on authentication based on eye-related biometric data, it is understood that other types of biometric data may also be used for authentication purposes in other embodiments. I want to be. Various embodiments discuss new conceptual schemes for conducting buying and selling in the context of augmented reality (AR) systems, as described below, but the techniques disclosed herein are optional. It should be understood that it can be used independently of existing and / or known AR systems. Therefore, the examples discussed below are for illustrative purposes only and the present invention should not be read as limited to AR systems.

With reference to FIG. 2A-2D, some common component options are illustrated. FIG. 2A-2D is followed by various systems, in some of the embodiments for carrying out the invention.
Subsystems, and components are presented to address the objectives of providing a high quality and comfortably perceived display system for human VR and / or AR.

As shown in FIG. 2A, the AR system user 60 is depicted wearing a frame 64 structure coupled to an AR display system 62 located in front of the user's eyes. The speaker 66 is coupled to the frame 64 in the configured configuration and is positioned adjacent to the user's ear canal (in one embodiment, another speaker (not shown) is positioned adjacent to the user's other ear canal. And provides stereo / moldable sound control). The display 62 is operably coupled to the local processing and data module by wire or wireless connectivity, etc. 68, and the local processing and data module is fixedly attached to the frame 64 or shown in the embodiment of FIG. 2B. Helmet or hat 80
Whether it is fixedly attached to, embedded in headphones, or removably attached to the fuselage 82 of the user 60 in a backpack configuration as shown in the embodiment of FIG. 2C.
Alternatively, as shown in the embodiment of FIG. 2D, the waist 84 of the user 60 in the belt coupling type configuration.
It can be mounted in various configurations, such as being removably mounted on the.

The local processing and data module 70 may include a power efficient processor or controller and a digital memory such as a flash memory, both of which may be utilized to assist in the processing, caching, and storage of the data. a) Captured from a sensor that can be operably coupled to a frame 64 such as an image capture device (camera, etc.), microphone, inertial measurement unit, accelerometer, compass, GPS unit, wireless device, and / or gyroscope. And / or b) Perhaps after such processing or reading, it is acquired and / or processed using the remote processing module 72 and / or the remote data repository 74 for passage to the display 62. The local processing and data module 70 provides a wired or wireless communication link or the like so that these remote modules (72, 74) are operably coupled to each other and available as resources to the local processing and data module 70. Via, it can be operably coupled (76, 78) to the remote processing module 72 and the remote data repository 74.

In one embodiment, the remote processing module 72 may include one or more relatively sophisticated processors or controllers configured to analyze and process data and / or image information. In one embodiment, the remote data repository 74 is the Internet or "
It may be equipped with relatively large digital data storage equipment that may be available through other networking configurations within the "cloud" resource configuration. In one embodiment, all data is stored locally and in a data module, where all calculations are done, which allows for fully autonomous use without any remote module.

As described with reference to FIGS. 2A-2D, the AR system continuously receives input from various devices that collect data about AR users and their surroundings. Here in Figure 3
The various components of an exemplary augmented reality display device will be described with reference to. It should be understood that other embodiments may have additional components.
Nevertheless, FIG. 3 provides a basic concept of the types of data that can be collected by various components and AR devices.

Here, with reference to FIG. 3, the schematic illustrates the coordination between the cloud computing asset 46 and the local processing asset (308, 120). In one embodiment, the cloud 46 assets are stored within a structure configured to be coupled directly to the user's head-mounted device 120 or belt 308 via wired or wireless networking (40, 42). Get local computing assets such as processor and memory configuration (
120, 308) operably coupled to one or both (radio is preferred for mobility and wired is preferred for certain high bandwidth or high data volume transfers that may be desired).
These user-local computing assets can also be operably coupled to each other via wired and / or wireless connectivity configurations 44. In one embodiment, in order to maintain a low inertia and small size head-mounted subsystem 120, the primary transfer between the user and the cloud 46 is via a link between the belt-based subsystem 308 and the cloud. Obtained, the head-mounted subsystem 120 primarily uses wireless connectivity such as ultra-wideband (“UWB”) connectivity, as currently employed in personal computing peripheral connectivity applications, for example. , Data tethered to a belt-based subsystem 308. Through the cloud 46, the AR display system 120 can interact with one or more AR servers 110 hosted in the cloud. Various AR servers 110 may have communication links 115 that allow the servers 110 to communicate with each other.

A world related to the user's current real or virtual location by using efficient local and remote processing coordination and the appropriate display device for the user, such as the user interface or the user "display device" or a variant thereof. Aspects can be transferred or "passed" to the user and updated in an efficient manner. In other words, the map of the world is, in part, resident on the user's AR system and, in part, continuously updated in a storage location that can be resident on cloud resources. A map (also referred to as a passable world model) can be a large database containing raster images, 3D and 2D points, parameter information, and other information about the real world. Maps become more and more accurate as AR users continuously capture more and more information about their real environment (eg, through cameras, sensors, IMUs, etc.).

In connection with the present disclosure, an AR system similar to that described in FIGS. 2A-2D is advantageously used to uniquely identify a user based on a set of biometric data tracked through the AR system. It provides unique access to the user's eyes that can be used. This unprecedented access to the user's eyes necessarily makes it suitable for a variety of applications. The AR device plays an important interaction with the user's eye to allow the user to perceive 3D virtual content, and in many embodiments, various biometrics associated with the user's eye (eg, eg). If binocular detachment movements, eye movements, rod and pyramidal patterns, eye movement patterns, etc.) are tracked, the resulting tracking data is advantageously described in more detail below. As such, it can be used in user identification and authentication for various transactions.

The AR device is typically adapted for a particular user's head and the optical components are aligned with the user's eyes. These configuration steps can be used to ensure that the user is provided with an optimal augmented reality experience without causing any physiological side effects such as headache, nausea, discomfort and the like. Thus, in one or more embodiments, the AR device can be configured for each individual user (both physical and digital) and calibrated for the user. In other scenarios, loosely adapted AR devices can be comfortably used by a variety of users. For example, in some embodiments, the AR device captures the distance between the user's eyes, the distance between the head-worn display and the user's eyes, the distance between the user's eyes, and the curvature of the user's forehead. do. All of these measurements can be used to provide a suitable head-worn display system for a given user. In other embodiments, such measurements may not be necessary to perform the identification and authentication functions described herein.

For example, referring to FIGS. 4A-4D, the AR device can be customized for each user. The user's head shape 402 may be considered when fitting the head-mounted AR system in one or more embodiments, as shown in FIG. 4A. Similarly, can the eye component 404 (eg, optics, structure for optics, etc.) be rotated or adjusted for user comfort, both horizontally and vertically, as shown in FIG. 4B. Or it can be rotated for user comfort. In one or more embodiments, as shown in FIG. 4C.
The head rotation point 406 set with respect to the user's head can be adjusted based on the structure of the user's head. Similarly, the interpupillary distance (IPD) 408 (ie, the distance between the user's eyes).
Can be compensated, as shown in FIG. 4D.

Advantageously, in user identification and authentication situations, this aspect of head-mounted AR devices is
The system can easily identify a user with a set of measurements for the user's physical characteristics (eg, eye size, head size, eye-to-eye distance, etc.) and allow the user to complete one or more commercial transactions. It is important because it already has other data that can be used to make it possible. In addition, the AR system may be readily detectable when the AR system is worn by a different AR user other than the user authorized to use the AR system. This allows the AR system to constantly monitor the user's eyes and therefore perceive the user's identity as needed.

In addition to the various measurements and calibrations made on the user, the AR device can be configured to track a set of biometric data about the user. For example, the system provides users with eye movements, eye movement patterns, blink patterns, binocular detachment movements, eye color, iris patterns, retinal patterns, fatigue parameters, eye color changes, focal length changes, and optical augmented reality experiences. Many other parameters that can be used in doing so can be tracked.

Referring to FIG. 5, one simplified embodiment of a suitable user display device 62 is shown to include a display lens 106 that can be mounted on the user's head or eyes by a housing or frame 108. Is done. The display lens 106 is positioned by the housing 108 in front of the user's eye 20 and bounces the projected light 38 into the eye 20 to form a beam while also allowing at least a portion of the light to pass from the local environment. May include one or more transparent mirrors configured to facilitate. In the illustrated embodiment, two wide-angle machine vision cameras 16 are coupled to the housing 108 to image the environment around the user. In one embodiment, these cameras 16 are dual capture visible / infrared light cameras.

The embodiments depicted also include a pair of scanning laser-formed corrugated surfaces with display mirrors and optics configured to project light 38 into the eye 20, as shown.
Optical projector module 18 (for depth) (eg, DLP, fiber scanning device (eg, DLP)
FSD), spatial light modulators such as LCD) are provided. The illustrated embodiment is also paired with an infrared light source 26, such as a light emitting diode "LED", configured to be capable of tracking the user's eye 20 and supporting rendering and user input. It is equipped with two small infrared cameras 24. The display system 62 further comprises X, Y, Z axis accelerometer capabilities and a magnetic compass and X, Y, Z axis gyroscope capabilities, preferably capable of providing data at relatively high frequencies such as 200 Hz. It features a sensor assembly 39. The system 62 depicted can also be configured to calculate real or near real-time user head postures from wide-angle image information output from the camera 16 ASICs (application specific integrated circuits), FPGAs (field programmable gate arrays), and the like. And / or A
It is equipped with a head posture processor 36 such as an RM processor (altitude reduction instruction set machine). The head posture processor 36 is operably coupled to the camera 16 and the rendering engine 34 (90, 92, 94; eg, via wired or wireless connectivity).

Another processor 32, which is configured to perform digital and / or analog processing and derive the attitude from the gyroscope, compass, and / or accelerometer data from the sensor assembly 39, is also shown. The embodiments depicted also feature a GPS37 subsystem to assist in posture and positioning.

Finally, the depicted embodiment is configured to provide the user's local rendering information for a view of the user's world, facilitating scanner operation and imaging into the user's eyes. It includes a rendering engine 34 that can be characterized by hardware that executes software programs. The rendering engine 34 includes a sensor orientation processor 32, an image orientation processor 36, and an eye tracking camera 24 so that the rendered light 38 is projected using the scanning laser array 18 in a manner similar to a retinal scanning display. , And operably coupled to the projection subsystem 18 (105, 94, 100 /
102, 104, i.e. via wired or wireless connectivity). Projected light beam 38
The wave front of the light 38 can be bent or focused to match the desired focal length of the projected light 38.

The small infrared eye tracking camera 24 can be used to track the eye and support rendering and user input (ie, where the user is looking, depth of focus, etc., as discussed below. , Binocular detachment can be used to estimate the depth of focus of the user). A GPS 37, a gyroscope, a compass, and an accelerometer 39 can be used to provide rough and / or fast attitude estimation. Camera 16 images and pose data, along with data from associated cloud computing resources, can be used to map the local world and share user views with virtual or augmented reality communities.

Much of the hardware in the display system 62 featured in FIG. 5 is depicted as being directly coupled to the display 106 and the housing 108 adjacent to the user's eye 20, although the hardware components depicted may be, for example, , As shown in FIG. 2D, may be mounted on or stored in other components such as the belt-mounted component 70.

In one embodiment, all components of the system 62 featured in FIG. 5 except the image attitude processor 36, the sensor attitude processor 32, and the rendering engine 34.
Directly coupled to the display housing 108, communication between the latter three and the remaining components of the system may be by wireless or wired communication, such as ultra-wideband. The housing 108 depicted is preferably head-mounted and can be worn by the user. It may also feature a speaker, such as one that is inserted into the user's ear and can be used to provide sound to the user.

Although the main components of a standard AR device have been described, it should be understood that an AR device can have many components that are configured to collect data from the user and its surroundings. For example, as mentioned above, some embodiments of AR devices are GP.
S Collect information and determine the location of the user. In another embodiment, the AR device comprises an infrared camera to track the user's eyes. In yet another embodiment, the AR device comprises a field camera, which may capture an image of the user's environment, which then maps the user's physical space (as described in FIG. 3, server 110). Can be used to configure (contained within one of), which allows the system to render virtual content in relation to the appropriate real object, as briefly described with respect to FIG. To enable.

With respect to the projection of the light 38 into the user's eye 20, in one embodiment, the small camera 24.
It can be used to measure the location where the center of the user's eye 20 is geometrically converged, which generally coincides with the position of the focus of the eye 20, i.e. the "depth of focus". The three-dimensional surface of all points that the eye approaches is called a "monoscopic locus". Focal lengths can take a finite number of depths or fluctuate infinitely. Binocular separation movement distance (vergence)
The light projected from the distance) appears concentrated in the target eye 20, while the light in front of or behind the binocular separation movement distance is blurred.

In addition, spatially coherent light with a beam diameter of less than about 0.7 mm
It was found to be properly degraded by the human eye, regardless of where the eye is focused. Given this understanding, binocular detachment movements, in order to generate an illusion of appropriate depth of focus,
Can be added by the small camera 24, the rendering engine 34 and the projection subsystem 18 focus on all objects on or near the monoscopic trajectory, and all other objects at varying degrees of defocus (ie,). It can be used to render (using intentionally generated blur). See-through optical waveguide optics configured to project coherent light into the eye are described by Lumus, Inc. Can be provided by suppliers such as. Preferably, the system 62 renders to the user at a frame rate of about 60 frames per second or higher. As mentioned above, preferably the small camera 2
4 can be utilized for eye tracking and the software can be configured to obtain not only binocular ectopic motion geometry, but also focal location clues to serve as user input. Preferably, such a system is configured with brightness and contrast suitable for daytime or nighttime.

In one embodiment, such a system preferably has a latency of less than about 20 milliseconds, an angular alignment of less than about 0.1 degrees, and a resolution of about 1 minute for visual object alignment. It is almost the limit of the human eye. The display system 62 may be integrated with a positioning system that may assist in positioning and orientation determination, including GPS elements, optical tracking, compasses, accelerometers, and / or other data sources. Positioning information can be used to facilitate accurate rendering within the user's view of the relevant world (ie, such information can be used.
Glasses will help them figure out where they are relative to the real world). Although the general components of an AR device have been described, additional embodiments relating to specific user identification and authentication for conducting commerce will be discussed below.

As discussed in some detail above, traditional models for conducting commerce tend to be inefficient and cumbersome, and often have the effect of preventing users from engaging in transactions. For example, suppose a user is in a department store. In the traditional model, the user physically enters the department store, selects goods, queues, waits for the cashier, payment information and /
Alternatively, you will be required to provide proof of identity and approve the payment. Even online shopping is probably less complicated, but it has its drawbacks. Although users do not have to be physically present at the store location and can easily select products of interest, payments still often require credit card information and authentication. However, with the advent of AR devices, traditional models of payment (eg cash, credit cards, monetary tokens, etc.) have become possible because AR devices can easily verify user identification and authenticate commerce. , Can be unnecessary.

For example, an AR user can hang out in a retail store and pick up a product. AR
The device may confirm the user's identity, determine if the user wants to make a purchase, and simply leave the store. In one or more embodiments, the AR device may interface with a financial institution that will transfer the amount from the user's account to the account associated with the retail store based on the confirmed purchase. Alternatively, in another example, an AR user may be viewing an advertisement for a particular brand of shoes. The user may indicate through the AR device that the user wants to buy shoes. The AR device can verify the user's identity and authenticate the purchase. At the back end, the order may be placed with a shoe brand retailer, who may simply ship a pair of desired shoes to the user. As illustrated by the example above, the AR device "knows" the identity of the user (and the AR device is typically built and customized for every individual user). Financial transactions are easily authenticated, thereby significantly reducing the frictional costs typically associated with conducting a sale.

More specifically, in one or more embodiments, the AR device may periodically perform user identification tests for privacy and security purposes. As discussed in some detail above, AR devices are typically customized for each user,
This periodic identification and authentication of the user is for security purposes, especially in the context of conducting commercial transactions, or the AR device is not being used by an unknown user and is linked to the AR user's account in the cloud. Necessary for privacy purposes to ensure that you are there. This application describes systems and methods for ensuring security for financial / commercial transactions where user identification and authentication are paramount. Similarly, these steps are equally important to ensure user privacy. actual,
These identification steps can be used prior to opening any personal / private user account (eg, email, social networks, financial accounts, etc.) through an AR device.

Other embodiments described herein can be used in the context of anti-theft management. For example, an AR device can identify a user and ensure that the AR device has not been stolen. If the AR device detects an unknown user, the AR device may immediately send the captured information about the user and the location of the AR device to the AR server. Or, in other embodiments, if it is detected that the AR device is being used by an unidentified person, the AR device will shut down completely and the AR will not be leaked or misused to prevent sensitive information from being leaked or misused. All content in the device's memory can be deleted automatically. These security measures can prevent the theft of the AR device because the AR device can capture many types of information about the wearer of the AR device.

In one or more embodiments, the embodiments described below may allow the AR device to remotely control the shopping robot. For example, once a user of an AR device is identified, the AR device will be launched through a network of specific stores or franchise stores.
It can connect to a shopping robot and communicate transactions with the shopping robot. Therefore, even if the user is not physically present in the store, the AR device can execute the transaction through the proxy once the user is authenticated. Similarly, many other security and / or privacy applications can be envisioned.

There are many ways to make an identification through tracked biometric data. In one embodiment, the tracked biometric data can be eye-related biometric data such as eye movement patterns, iris patterns, binocular detachment movement information, and the like. In short, rather than requiring the user to remember the password or present some type of identification / proof.
The AR device automatically validates the identification through the use of tracked biometric data. Given that the AR device has constant access to the user's eyes, it is expected that the tracked data will provide very accurate and personalized information about the user's identity, and that information. Can be considered a unique user signature. Track biometric data and use the tracked biometric data to authenticate users with one or more external entities (eg, financial institutions, distributors, etc.) before delving into the details of different methods. A system architecture for interacting AR devices will be provided.

Here, with reference to FIG. 6, the overall AR system architecture is illustrated. The AR system architecture includes a head-mounted AR device 62, a local processing module 660 of the AR device 62, a network 650, an AR server 612, and a financial institution 620.
And one or more distributors (622A, 622B, etc.).

As mentioned above (see Figure 5), the head-worn AR device 62 has many sub-components, some of which capture and / or track information associated with the user and / or the user's surroundings. It is configured to do. More specifically, in one or more embodiments, the head-worn AR device 62 may include one or more eye tracking cameras. The eye tracking camera can track the movement of the user's eyes, binocular detachment movement, and the like. In another embodiment, the eye tracking camera may be configured to capture a picture of the user's iris. In other embodiments,
The head-mounted AR device 62 may include other cameras configured to capture other biometric information. For example, the associated camera may be configured to capture an image of the user's eye shape. Alternatively, in another example, the camera (or other tracking device) may capture data about the user's eyelashes. Tracked biometric information (eg, eye data, eyebrows data, eye shape data, eye movement data, head data, sensor data, voice data, fingerprint data, etc.) can be transmitted to the local processing module 660.
As shown in FIG. 2D, in one or more embodiments, the local processing module 660
It can be part of the belt pack of the AR device 62. Alternatively, in other embodiments, the local processing module may be part of the housing of the head-mounted AR device 62.

As shown in FIG. 6, the user 680 head-worn AR device 62 interfaces with the local processing module 660 to provide captured data. In one or more embodiments, the local processing module 660 is a processor 664 and other components 6 that allow the AR system to perform various computing tasks.
52 (for example, memory, power supply, telemetry circuit, etc.) is provided. Significantly in the present disclosure, the local processing module 660 also comprises an identification module 614, which can identify the user based on the information tracked by one or more tracking devices of the head-mounted AR device 62.

In one or more embodiments, the identification module 614 comprises a database 652, stores a set of data, and uses it to identify and / or authenticate the user. For example, in the illustrated embodiment, the database 652 includes a mapping table 670 that can store a given set of data and / or a given authentication detail or pattern. In one or more embodiments, the captured data can be compared to predetermined data stored in the mapping table 670 to determine the identification of the user 680. Similarly, the database 652 may include other data that will be used in identifying / authenticating the user 680.
In one or more embodiments, the database 652 may store one or more eye tests and verify user identification, as will be described in more detail below.

In one or more embodiments, the local processing module 660 is a cloud network 6
Communicate with the AR server 612 through 50. Although not shown in FIG. 6, the AR server 612 includes many components / circuits that are important for providing a realistic augmented reality experience to the user 680. In summary, the AR server 612 comprises a physical world map 690 that is frequently referenced by the local processing module 660 of the AR device 62 to render virtual content in relation to real world physical objects. There is. Therefore, the AR server 612 is built on the information captured through a large number of users to build the ever-growing map 690 in the real world. In other embodiments, the AR server 612 may simply host the map 690, which can be built and maintained by a third party.

In one or more embodiments, the AR server 612 may also host an individual user's account, to which the user's privately captured data is supplied. This captured data is 1
In one or more embodiments, it may be stored in database 654. In one or more embodiments, the database 654 contains user information 610, historical data 615 for user 680.
, User preference 616, and other entity authentication information 618 may be stored. In fact, AR
Other embodiments of the system may include many other types of information that are personal to the user. User information 610 may include a set of personal biographies (eg, name, age, gender, address, location, etc.) in one or more embodiments.

Historical data 615 for user 680 may refer to previous purchases and / or transactions made by the user. In one or more embodiments, the user preference 616 presents a set of interests (eg, shopping, activities, travel, etc.) and / or purchase preferences for the user (eg, accessories, brands of interest, shopping categories, etc.). Can include. In one or more embodiments, AR user behavior data can be used to inform the system of user preferences and / or purchase patterns. Other entity credentials 618 may refer to a user's credentials to prove that the user has successfully authenticated access to an external account (eg, bank credentials, account credentials for various websites). etc)
In one or more embodiments, the data captured through the AR device 62, the data tracked through past activity, the buying and selling data associated with the user, etc., to recognize and / or understand the user's behavioral patterns. Can be analyzed. These features
In one or more embodiments, it may be performed by a third party in a privacy and security-friendly manner.

Database 654 may also store other entity credentials 618 that allow the AR server 612 to communicate with a particular financial institution and / or third party entity. For example, the other entity authentication information 618 may point to the user's bank information (eg, bank name, account information, etc.). This information can then be used to communicate with financial information, third party entities, distributors, etc.

In one or more embodiments, the AR server 612 may communicate with one or more financial institutions 620 to complete the transaction. The financial institution may have the user's financial information. In one or more embodiments, the financial institution may perform a second verification of the user's credentials for security purposes. When the user is authenticated, the AR server 612 is authenticated and can communicate with the financial institution 620. If the user is authenticated for a particular purchase of goods 630, the financial institution 620
Once the user is authenticated, it communicates directly with one or more distributors (622A, 622B, etc.) and
The amount can be transmitted directly to the seller. In other embodiments (not shown), the AR server 612
Can communicate directly with the seller and may communicate data about one or more purchases.

In some embodiments, user 680 AR to proceed with one or more financial transactions.
It should be understood that it may not be necessary to connect to server 612. For example, in some embodiments, the AR device 62 may allow "offline browsing" of multiple e-commerce sites, etc., and the user 680 may select one or more products of interest through an offline ID. Can be possible. The financial institution 620 or distributor 622 may have a random number generator generated for that particular transaction, which can be verified later when the AR device 62 is connected to network 650. In other words, the AR device 62 is AR
Even when not connected to the server, the system may review the transaction offline and then use additional information (eg, randomly generated numbers) to later activate the purchase.
This allows the AR device 62 to participate in the required commerce even if the user is not currently connected to the AR server 612 and / or the financial institution or distributor.

In one or more embodiments, the distributor (622A, 622B, etc.) has a pre-established relationship with the AR server 612 and / or financial institution 620 that allows for this new conceptual scheme of making purchases through the AR device 62. Can have. It should be understood that the aforementioned embodiments are provided for illustrative purposes only, and other embodiments may include more or fewer components.

With reference to FIG. 7, an exemplary flow chart for conducting a commercial transaction through the AR device 62 is provided. At 702, input regarding the transaction may be received. For example, a user may explicitly indicate that he or she is interested in purchasing an item (eg, through commands, gestures, etc.). In another embodiment, the AR system may propose a purchase to the AR user and receive confirmation from the user based on past purchases, user interests, and the like. In yet another embodiment, the AR system may assess interest based on a "heat map" of various products. More specifically, since the AR device 62 knows where and when the user is looking, the AR device 62 allows the user to determine various virtual and virtual interests in the product. / Or it may be possible to determine how long you have been looking at the real object. For example, if the user is looking at a virtual advertisement of a specific brand, the AR device 6
2 can measure the user's interest by determining how long the user has been looking at a particular product. In one or more embodiments, the AR system may generate a heatmap based on the amount of time one or more users have been viewing a particular product. If the heatmap shows interest in a particular product (for example, the amount of time spent looking at a particular product exceeds a given threshold amount of time), AR
The device 62 may request confirmation from the AR user regarding the purchase of the product.

At 704, the AR device 62 may perform a user identification protocol. There can be many types of user identification protocols, as described in more detail below. In one or more embodiments, the AR device 62 may simply require a "password" based on eye movement to determine if the user is validated. In another embodiment, the AR device 62 may capture a picture of the user's iris to see if the user is a valid user of the AR device 62 (and the account linked to the AR device 62). In another embodiment, the AR device 62 may monitor the continuity of the AR device 62 that remains on the user's head (eg, if the user has not removed the AR device 62 at all, the user is the same. Probability is high). In one or more embodiments, the AR device 62 periodically captures an iris image based on a user identification protocol, or the user uses the AR device 62.
Testing can be done periodically to ensure that you are a verified user of. As discussed herein, there are many ways to identify a user through biometric data, and some exemplary methods will be further described below.

In some embodiments, the identification protocol can be a constant identification of the user (eg, eye movement patterns, skin contact, etc.). In other embodiments, the identification protocol is simply
It can be a one-time identification (through any identification method). Thus, in some embodiments, once the AR system identifies a user, the same user will be identified unless an intervention event occurs (eg, the user removes the AR device 62, disrupts network connectivity, etc.). It may not need to be done. At 705, the AR device 62 may determine whether the identification protocol requires the capture of biometric data. If the user protocol requires that the biometric data be captured, the AR device may capture the biometric data. Otherwise, the AR device 62 may proceed with user identification through a non-biometric capture identification method.

At 706, biometric data can be captured based on the user identification protocol. For example, if the user identification protocol is an eye test for detecting known patterns, the AR device 62 may track the movement of the user's eye through one or more eye tracking cameras. Captured movements can be correlated with "passwords" or signature eye movements to determine if the user is validated. Alternatively, if the user identification protocol is iris capture, the image of the iris can be captured and correlated with a known image of the user. Alternatively, each time the AR user wears the AR device 62, an iris capture or eye test may be performed to verify the user's identification. It should be understood that the biometric data can be eye-related or other types of biometric data in some embodiments. for example,
The biometric data can be audio in one or more embodiments. In one or more embodiments, the biometric data can be eyelash-related data or eye shape data. Any type of biometric data that can be used to uniquely identify a user from other users can be used.

At 708, biometric data can be compared with predetermined user identification data to identify the user. Alternatively, if the user identification does not require biometric data, the AR device 62 determines, for example, that the user has not removed the AR device 62, and thus the user is identical to the previously identified user. Can be shown. If the user is identified, the AR device 62 proceeds to 710 and transmits the information to one or more financial institutions.

If the user is not identified, the AR device 62 performs another user identification protocol and
Or otherwise, it can prevent the user from making a transaction. If the user is identified, data about the desired product may be transmitted to the cloud and financial institutions at 710. For example, according to the above example, information about the desired shoe (eg, product number, desired quantity, information about the user, shipping address, user account, etc.) may be communicated to the seller and / or financial institution.

At 712, the AR system may receive confirmation from the financial institution that the payment has been completed and / or approved. At 714, a confirmation message may be displayed to the user to confirm that the purchase is complete.

As mentioned above, in one or more embodiments, one approach to identifying users (for financial transactions and other purposes) for verification purposes is to perform user identification tests periodically. obtain. In one or more embodiments, the user identification method utilizes eye-related data.
The user identification test can be completed. Since the AR device 62 is equipped with an eye tracking camera that continuously tracks the movement of the user's eyes, known patterns of eye movement can be used as eye tests to recognize and / or identify the user. For example, passwords can be easily copied or stolen, but it is difficult to replicate the eye movements or other physiological properties of other users, making it easy to identify unauthorized users of the AR device 62. obtain.

In one or more embodiments, during the configuration of the AR device 62, the system may use the user's input to configure a known pattern of eye movement unique to the user (ie, similar to an eye password). This known pattern of eye movement can be memorized and correlated each time a user identification protocol is performed.

Here, with reference to embodiment 800 of FIGS. 8A and 8B, an exemplary eye pattern 802 of the user's eye 804 is provided. As shown in FIG. 8A, the AR device 806 may track the user's eye movement through an eye tracking camera (not shown) and correlate the pattern with a known pattern of eye movement (ie, eye password). .. If the eye movement pattern is close to a known pattern (within a threshold), the AR device 806 may allow the user to make a transaction. As shown in FIG. 8A, the user can move his eyes in the pattern shown. For illustration purposes, lines (802) are drawn to represent eye patterns. Of course
In reality, there are no lines and the eye tracking device will simply track such movements and convert it to the desired data format.

In one or more embodiments, a grid 904 similar to that shown in FIG. 8B is used to determine if the tracked eye pattern correlates with a predetermined known pattern of eye movement.
Can be used. It should be understood that other such techniques can be used as well. Through the use of grid 904, by dividing the available space into discretized areas, it can be facilitated to determine if the tracked eye pattern 802 is most similar to a given pattern 902. For example, as shown in FIG. 8B, the tracked eye pattern 802.
Follows a given pattern 902 in any way (as indicated by the thick lines connecting the centers of each grid square associated with a given pattern). FIG. 8B shows the grid 90.
Representing a fairly simplified version of 4, it should be understood that the size of each grid can be reduced for more accurate determination.

In the illustrated embodiment, the pattern can be recognized if the tracked eye movement targets an area of a predetermined grid square. In other embodiments, the majority of grid squares,
It may need to be hit before the user is considered to have passed the user identification test. Similarly, other such thresholds can be devised for various eye movement tracking protocols. In one or more embodiments, blink patterns can be utilized as well, as described above. For example, rather than utilizing an eye movement pattern, the eye password can be a series of blinks or blinks combined with a movement to track a user's signature.

Here, with reference to FIG. 9, an exemplary process for detecting an eye movement signature is described. 90
In 2, the eye movement test can be initiated. For example, the AR user may indicate that he wants to purchase the goods, or the AR user may remove the AR devices 62,806,
The mounting of the AR devices 62 and 806 can be resumed. In another embodiment, the eye movement test may be performed cyclically for security purposes.

At 904, eye movement patterns can be tracked and received. For example, a virtual display screen may display a "password entry" command, which may trigger the user to form a known pattern with his eyes.

At 906, the tracked eye movements can be transformed into a particular data format. For example, returning to the grid approach and referencing, the data may indicate the coordinates of the grid hit by eye movement. Many other approaches can be used as well.

At 908, the transformed data can be compared with predetermined data of known signature eye movement patterns. At 910, the AR system can determine if the tracked eye movement matches a predetermined pattern within a threshold. In 912, if it is determined that the eye pattern does not match the known eye pattern within the threshold, the user may fail the test and be prevented from making a purchase or need to take the test again. could be. In 914, the eye pattern is
If it is determined to match a known eye pattern within the threshold, the user passes the test and
You may be allowed to make a purchase.

In yet another approach, rather than performing an eye test, the AR system periodically captures pictures of the AR user's eyes and compares the captured images of the user's eyes with known information. Can perform eye identification. In one or more embodiments, the AR devices 62,806 may require the user to stare at a particular virtual object presented to the user when the user intends to make a purchase. This allows the user's eye to rest and images of the user's eye can be captured and compared. If the photo of the eye correlates with a known photo of the user's eye, the AR user may be allowed to make a purchase. For more details on eye identification techniques, see "DEVICES, METHODS A" under patent attorney reference number ML3002.80.
ND SYSTEMS FOR BIOMETRIC USER RECOGNITIO
Provided in a co-pending application No. 62 / 159,593 entitled "N UTILIZING NEURAL NETWORKS".

This feature is advantageous because the AR system generally needs to know where the user's eyes are looking (or "seeing") and where the user's eyes are focused. Can be used for identification purposes. Thus, in various embodiments, the head-mounted display (“HMD”) component features one or more cameras that are directed to capture image information about the user's eyes. In one configuration, such as those depicted in FIG. 5, each user's eye has a camera focused on it, as well as three or more with a known offset distance to the camera for inducing a flash onto the surface of the eye. LED (in one embodiment,
Can have (just below the eye) as shown.

The presence of three or more LEDs with a known offset to each camera makes it possible to determine the distance from the camera to each flash point in 3D space by triangulation. Using at least three flash points and a substantially spherical model of the eye, the system can infer the curvature of the eye. Using known 3D offsets and orientations to the eyes, the system is accurate (eg, images) of the iris or retina for use to identify the user.
Alternatively, an abstract (eg, gradient or other feature) template can be formed.
In other embodiments, other characteristics of the eye, such as patterns of veins in and above the eye, can also be used to identify the user (eg, with an iris or retinal template).

In one approach, the iris image identification approach can be used. The pattern of muscle fibers within the iris of the eye forms a stable and unique pattern for each individual, including spots, grooves, and rings. Various iris features can be easily captured using infrared or near-infrared imaging as compared to visible light imaging. The system can convert captured iris features into identification codes in many different ways. The goal is to extract a sufficiently abundant tissue structure from the eye. With sufficient degrees of freedom in the collected data, the system can theoretically identify a unique user.

In another approach, retinal image identification can be used as well. In one embodiment, the HMD
Features a diffractive display driven by a laser scanner steered by steerable fiber optic cables. This fiber optic cable also visualizes the inside of the eye and
It can also be used to image visual receptors (rod and pyramidal cells) as well as the retina, which has a unique pattern of blood vessels. These rod and pyramidal cells can also form patterns that are unique to each individual and can be used to uniquely identify each individual.

For example, each person's dark and clear blood vessel patterns are unique, such as applying a gradient operator to a retinal image and counting elevation transitions within a standardized grid centered on the retina. It can be converted to a "dark-bright" code by a technique.

Therefore, the system of this subject, with a high degree of accuracy and accuracy, by comparing the user characteristics captured or detected by the system with the known baseline user characteristics for the approved users of the system. Can be used to identify.

In yet other embodiments, the curvature / size of the eye can be used as well. For example, this information may aid in the identification of users because the eyes of different users are similar but not exactly the same. In other embodiments, transient biometric information can be collected when the user is under stress and correlated with known data. For example, the user's heart rate
Can be monitored, whether the user's eyes are producing a film of water, whether the eyes are approaching and focusing together, breathing patterns, blink rates, pulse rate, etc. also confirm the user's identification And / or can be used to disable.

In yet another embodiment, to confirm the user's identity (eg, if misidentification is suspected), the AR system captures information through the AR device (eg, through the field camera of the AR devices 62,806). Correlate the image of the surrounding environment), and the user
It can be determined whether you are looking at the same scene that correlates to a location as derived from GPS and environmental maps. For example, if the user is considered to be at home, the AR system can verify by correlating known objects at the user's home with what they are seeing through the user's field camera.

The AR / user identification system described above provides a very secure form of user identification.
In other words, the system can be utilized to determine the user with a relatively high degree of accuracy and accuracy. The system can be used to identify users with a very high degree of certainty and on a continuous basis (eg, using periodic monitoring), requiring a separate login. It can be used to enable various financial transactions without. One approach to ensuring that the user identification system is very accurate is through the use of neural networks, patent attorney reference number ML30028.
It is explained in more detail in the co-pending application No. 62 / 159,593 under 00.

Here, with reference to FIGS. 10A-10I, an exemplary process flow 1000 using biometric data to carry out a transaction is illustrated. As shown in FIG. 10A
User 1002 wearing the AR device 1004 walks into the store. While in the store, user 1002 may see a pair of shoes 1006 that may be of interest to purchase.

Here, with reference to FIG. 10B, an exemplary view of the shoe visible to user 1002 through the AR device 1004 is shown. Upon detecting that the user's gaze is focused on the pair of shoes 1006, the AR device 1004 looks up details about the pair of shoes 1006 (eg, through a product catalog synchronized with the pair of shoes 1004). , Details can be displayed as virtual content 1008. With reference to FIG. 10C here, the AR device 1004 may determine whether the user wants to purchase consumption by displaying the virtual content 1010. User 1002 may confirm or reject through any form of user input (eg, gesture, voice, eye control, etc.).

Referring here to FIG. 10D, assuming that the user decides to purchase, AR device 1
004 may request a password through the virtual content 1012. At this point, FIG.
As shown in 0E, user 1002 may begin to generate eye signature 1016. In one or more embodiments, the virtual grid 1014 may be presented to the user to assist in moving the eyes in a particular manner.

As shown in FIG. 10F, the input signature 1016 may be received by the AR system 1004 and compared with a predetermined signature to determine if the user is an authenticated user. If the user is authenticated, the AR device 1004 will have the AR device 1004 as shown in FIG. 10G.
Data about the desired product can be transmitted through the network 1018 to the AR server 1020 to the distributor 1024 and the financial institution 1022. Based on the confirmation received from the AR server 1020, the financial institution 1022 may transmit the appropriate amount to the distributor 1024.

As shown in FIG. 10H, when the transaction is confirmed, the AR device 1004 is set to the shoe 100.
Virtual content 1026 confirming the purchase of 6 may be displayed. Upon receipt of the confirmation, the user 1002 may leave the store with the desired shoe 1006, as shown in FIG. 10I.

It should be understood that the process flow of FIGS. 10A-10I is presented herein for illustrative purposes only and should not be read as a limitation, but merely represents an exemplary embodiment. Many other embodiments can be envisioned as well. For example, in one or more embodiments
Rather than requiring a "password" (eg, FIG. 10D), the AR system requires the user to stare at the virtual dots on the screen and the user's eyes (eg, retina, iris, eye shape, etc.). Image can be captured. This image can then be correlated with a known image of the user's eye, confirming the user's identification. Once the user's identity is confirmed, the AR system will see Figure 1.
Information can be transmitted to distributor 1024 and financial institution 1022 as shown in 0G.
Similarly, many other similar embodiments can be envisioned.

As mentioned above, it should be understood that such authentication and payment systems make transactions much easier than traditional payment models. Rather than a long and painstaking round trip to a department store, shopping becomes a "playground" experience, where users simply walk into the store, pick up any number of products, and simply leave the store. I can go. AR system
It undertakes most of the payment details, while requiring only simple and unobtrusive identification checks based on easily tracked biometric data. As mentioned above, the identification check does not require any user action at the time of purchase, according to some embodiments.

As discussed in detail above, traditional passwords or sign-up / login codes can be excluded from individual secure transactions using the AR / user identification systems and methods described above. The system of the subject can pre-identify / pre-authenticate the user with a very high degree of certainty. In addition, the system can use periodic monitoring to maintain user identification over time. Thus, the identified user can have instant access to any site after being notified about the terms of that site (which can be displayed to the user as an overlaid user interface item).
In one embodiment, the system may create a set of standard provisions pre-determined by the user so that the user instantly understands the conditions on the site. The site is this condition (eg
If you do not comply with a set of standard provisions), the system in this subject may not automatically allow access or transactions there.

In addition, the AR / user identification system described above can be used to facilitate "small transaction". Small transactions that generate only very small debits and credits in a user's financial account are typically about a few cents or less than a cent. On a given site, the system of the subject is not only the content viewed or used by the user, but also the time (short browsing can be free, but beyond a certain amount will be charged). Can be configured to confirm. In various embodiments, news articles can be 1/3 cents, books can be charged 1 penny per page, and music can be 10 per viewing.
Cent etc. In another embodiment, the advertiser may pay the user 0.5 cents for selecting a banner ad or conducting a survey. The system may be configured to allocate a small percentage of the transaction fee to the service provider.

In one embodiment, the system can be used to create a dedicated small transaction account that can be controlled by the user, and the funds associated with the small transaction are aggregated and the user's more traditional financial account as a predetermined meaningful amount. Distributed to / from (eg, online bank accounts). Small trading accounts can be cleared or deposited at regular intervals (eg, quarterly) or in response to certain triggers (eg, when a user spends more than a few dollars on a particular website).

Small-value transactions are typically impractical and cumbersome in traditional payment conceptual schemes, but as described in the payment techniques described herein, transactions are near-instantaneous user identification and authentication. The ease that arises through eliminates many of the obstacles typically associated with small-value transactions. This can open up new means of monetization for future businesses. For example, it may be easier to monetize music, books, advertisements and the like. Users can hesitate to pay $ 1 for news articles, but they may not hesitate too much for articles that cost a fraction of a cent. Similarly, given that it is significantly easier to carry out these transactions (small and high), advertisers, publishers, etc. are more likely to develop content for different types of payment schemes. Can increase. Therefore, the AR system facilitates both payment and content delivery, thereby making both the front-end and back-end processes relatively simple.

The system and functionality of the subject can be provided by companies focused on augmented reality, and the user's identity is so reliably and securely known that the user has instant access to that account. , Amounts, spending, spending rates, and 3D views of graphical and / or geographical maps of that spending may be provided. Such users may be able to instantly adjust spending access, such as turning spending (eg, small transactions) off and on.

Due to high expenditures (ie, a few penny or a few dollars, not a percentage of penny), various embodiments may be facilitated using the system configuration of the subject.

The user may use the system to order perishables for delivery to its tracked location or user-selected map location. The system can also notify the user when a delivery will arrive (eg, by displaying a video of the delivery taking place within the AR system). With AR telepresence, a user is physically located in an office away from their home, but a courier enters the home, appears to the courier by avatar telepresence, and the courier delivers the product. You can see the delivery, then confirm that the courier has left, and lock the door of your home with avatar telepresence.

Optionally, the system may memorize the user's product preferences and alert the user to a sale or other promotion related to the user's preferred product. For these high spending embodiments, the user can review the account overview, all statistics for the account, and purchasing patterns, thereby facilitating a shopping comparison before placing the order.

Since the system can be used to track the eyes, it can also allow "at a glance" shopping. For example, a user looks at an object (eg, a robe in a hotel) and
Suppose you say, "I want this if the amount in my account exceeds $ 3,000." The system will make a purchase when certain conditions (eg, the account balance exceeds $ 3,000) are met.

As mentioned above, in one or more embodiments, the iris and / or retinal signature data can be used to secure communication. In such embodiments, the system of the subject only allows the user to authenticate based on one or more dynamically measured iris and / or retinal signatures. It may be configured to allow it to be selectively transmittable and viewable on a trusted secure hardware device that allows access. Since the AR system display device projects directly onto the user's retina, only the intended recipient (identified by the iris and / or retina signature) may be able to view the protected content, and in addition.
As the viewing device actively monitors the user's eye, the dynamically read iris and / or retinal signature can be recorded as proof that the content was actually presented to the user's eye (eg, perhaps). As a form of digital reception performed by verification actions, such as performing a sequence that requires eye movement).

Spoofing detection can rule out attempts using previous recordings of retinal images, static or 2D retinal images, generated images, etc., based on a model of expected natural variation. Unique criteria / watermarks can be generated and projected onto the retina to generate unique retinal signatures for auditing.

The financial and communication systems described above are provided as examples of various general systems that can benefit from more accurate and precise user identification. Therefore, the use of AR / user identification systems as described herein is not limited to the disclosed financial and communication systems, but rather rather.
Applicable to any system that requires user identification.

Various exemplary embodiments of the invention are described herein. In a non-limiting sense, these examples are referred to. They are provided to illustrate the broader and applicable aspects of the invention. Various modifications can be made to the invention described and equivalents can be replaced without departing from the true spirit and scope of the invention. In addition, many modifications can be made to adapt a particular situation, material, material composition, process, process action, or step to the object, spirit, or scope of the invention. Moreover, as will be appreciated by those skilled in the art, each of the individual variants described and illustrated herein will be of some other embodiment without departing from the scope or spirit of the invention. It has individual components and features that can be easily separated from or combined with any of the features. All such amendments are intended to be within the claims associated with this disclosure.

The present invention includes methods that can be performed using the subject device. The method may include the act of providing such a suitable device. Such provision may be made by the end user. In other words, the act of "providing" simply acquires, accesses, approaches, positions, configures, launches, powers on, and provides the end user with the essential device in the subject method. Or require it to act differently. The methods described herein can be performed in any order in which the events are described, which is logically possible, and in the order in which the events are described.

Illustrative aspects of the invention are described above, along with details regarding material selection and manufacture. With respect to other details of the invention, they are understood in connection with the patents and publications referenced above, and are generally known or can be understood by those skilled in the art. The same may be true with respect to the method-based aspects of the invention in terms of additional actions that are generally or logically adopted.

In addition, while the invention is described with reference to some examples that optionally incorporate various features, the invention is described and directed as considered for each variant of the invention. It is not limited to things. Various modifications may be made to the invention described and are included for some simplification, as described herein, without departing from the true spirit and scope of the invention. (Whether or not) can be replaced. In addition, if a range of values is provided, all intervening values between the upper and lower bounds of the range, and any other provisions or intervening values within that defined range, are included within the invention. Please understand that.

Also, any optional features of the embodiments of the invention described are described and claimed independently or in combination with any one or more of the features described herein. To get is considered. References to singular items include the possibility that the same plural item exists. More specifically, as used herein and in the claims associated with this specification, "one (" a "," an ")", "said", and " The singular form "the (the)" includes plural directive references unless otherwise specified. In other words, the use of articles allows for "at least one" subject item in the above description as well as in the claims associated with the present disclosure. Moreover, such claims are
Note that it can be drafted to exclude any voluntary elements. Therefore, this statement is antecedent for the use of exclusive terms such as "simply", "only", and equivalents, or the use of "negative" restrictions in connection with the description of the elements of the claim. The purpose is to fulfill the function as.

Without using such exclusive terms, the term "equipped" in the claims associated with this disclosure is such that a given number of elements are listed or featured in such claims. It shall be possible to include any additional elements, regardless of whether the addition of may be considered as a transformation of the nature of the elements described in such claims. Except as specifically defined herein, all technical and scientific terms used herein are generally understood as broadly as possible while maintaining the validity of the claims. It is given the meaning of being.

The scope of the present invention is not limited to the specification of the examples and / or the subject matter provided, but rather to the scope of the claims associated with the present disclosure.

Claims (48)

A method of initiating a transaction through an AR system that includes an augmented reality (AR) device and a local processing module.
To capture eye-related biometric data of a user wearing the AR device, the eye-related biometric data is a component of the AR device and to track the eye-related biometric data. Captured using at least one light source and at least one camera that can work cooperatively with
Analysis of eye-related biometric data by the local processing module by the local processing module executing an eye movement-based user identification protocol in response to identifying a real-world object in the direction of the user's gaze. The gaze is determined based on data collected by the at least one camera and analyzed by the local processing module, at least in part, to determine the identification of the user. Eye movement-based user identification protocols
The predetermined eye signature movement pattern is to display an instruction prompting the user to virtually draw an eye movement-based password by moving the user's gaze according to a predetermined eye signature movement pattern. A real-world object that is unique to the user and in the direction of the user's gaze, determined before identifying the display of the instruction, and not displayed to the user as part of displaying the instruction. When,
The user tracks the user's non-linear and multi-directional eye movements by moving the user's gaze in response to the displayed command, wherein the non-linear and multi-directional virtual line is the said. The user represents the user's eye movement pattern that is virtually drawn by the user moving the user's gaze in response to the displayed command, and the tracked non-linear and multidirectional eye movement is the AR device. Is captured through at least one camera of the AR device while being worn on the user's head.
Including, and
Comparing the non-linear and multi-directional virtual line representing the user's eye movement pattern virtually drawn by the user based on the tracked non-linear and multi-directional eye movement with the predetermined eye signature movement pattern. When,
Determining that the non-linear and multi-directional virtual line corresponds to the predetermined eye signature movement pattern based on the non-linear and multi-directional virtual line within a predetermined threshold of the predetermined eye signature movement pattern. When,
In response to determining that the non-linear and multi-directional virtual line corresponds to the predetermined eye signature movement pattern, the local processing module communicates to initiate a transaction to purchase the object. To transmit, the communication is transmitted through at least one network to a remote server that authenticates the user for the transaction based on the determined identification.
Including methods. The method of claim 1, wherein the remote server further comprises transmitting a set of data relating to the transaction through a second network to a computer of a financial institution. The method of claim 1, wherein the eye-related biometric data further comprises an iris pattern of the user's eye on the AR device. The method of claim 1, further comprising capturing biometric data, including voice recordings of the user, by the speaker of the AR device. The method of claim 1, wherein the eye-related biometric data further comprises a retinal signature of the user's eye on the AR device. The method of claim 1, further comprising capturing biometric data, including properties associated with the user's skin, with the AR device. The method of claim 1, wherein the eye-related biometric data further comprises a blink pattern of the user's eyes. The method of claim 1, wherein the AR device is individually calibrated to the user's head and eyes. The method of claim 1, wherein determining the identification of the user is at least partially based on a known iris pattern. The method of claim 1, wherein determining the identification of the user is at least partially based on a known retinal pattern. The method according to claim 1, wherein the transaction to be started is a commercial transaction. The remote server further comprises communicating the user's authentication to the computer of the financial institution associated with the user, the computer of the financial institution at least partially to the authentication received from the remote server. The method of claim 1, wherein payment is made on behalf of the user based on. 12. The method of claim 12, wherein the computer of the financial institution transmits the payment to the computer of one or more sellers indicated by the user. When the local processing module detects an interruption event associated with the AR device,
The AR device captures new biometric data from the user in order to reauthenticate the user based on the detected interruption event and the new biometric data at least in part.
The method according to claim 1, further comprising. 14. The method of claim 14, wherein the interruption event is detected by the local processing module, at least partially based on the removal of the AR device from the user's head. 14. The method of claim 14, wherein the interruption event is detected by the local processing module, at least in part, based on the loss of connectivity of the AR device to the network. The user's gaze is detected by the local processing module, at least in part, based on a heatmap associated with the user's gaze in the real-world object presented through the AR device. The method described in. 1. The non-linear and multi-directional virtual line representing the user's eye movement pattern resulting from the user pattern virtually drawn by the user moving the user's gaze comprises a plurality of linear sections. The method described. The non-linear and multi-directional virtual line is based on the non-linear division of the non-linear and multi-directional virtual line within a predetermined threshold of the corresponding linear division of the predetermined eye signature movement pattern. 18. The method of claim 18, which is determined to correspond to. The method of claim 1, wherein during the display of the real-world object in the direction of the user's gaze, the instructions are displayed and the non-linear and multi-directional eye movement of the user is tracked. The predetermined eye signature movement pattern informs the user while the user tracks the non-linear and multidirectional eye movement of the user by moving the user's gaze in response to the displayed command. The method of claim 1, which is not displayed. Further comprising transforming the format of the non-linear and multi-directional virtual line data, the transformed data is the non-linear and multi-directional virtual line in which a plurality of discrete areas of the virtual grid are drawn by the user. The method of claim 1, wherein the method was crossed by the non-linear and multi-directional eye movement resulting in. A computerized augmented reality (AR) display system
An AR device, wherein the AR device is structured to be worn on the user's head of the AR device, and the AR device can operate cooperatively to capture eye-related biometric data. An AR device with at least one light source and at least one camera,
With a local processing module that communicates with the AR device
With
The AR device can be operated by the at least one light source and the at least one camera to track the user's eye-related biometric data while worn by the user.
The local processing module
Analyzing the eye-related biometric data and
Identifying a real-world object in the direction of the user's gaze, which gaze is determined based on data collected by the at least one camera and analyzed by the local processing module. When,
Identification of the user based at least in part on the analysis of eye-related biometric data by execution of an eye movement-based user identification protocol in response to identifying the real-world object in the direction of the user's gaze. The eye movement-based user identification protocol is to determine.
The predetermined eye signature movement pattern is to display an instruction prompting the user to virtually draw an eye movement-based password by moving the user's gaze according to a predetermined eye signature movement pattern. A real-world object that is unique to the user and in the direction of the user's gaze, determined before identifying the display of the instruction, and not displayed to the user as part of displaying the instruction. When,
The user tracks the user's non-linear and multi-directional eye movements by moving the user's gaze in response to the displayed command, wherein the non-linear and multi-directional virtual line is the said. The user represents the user's eye movement pattern that is virtually drawn by the user moving the user's gaze in response to the displayed command, and the tracked non-linear and multidirectional eye movement is the AR device. Is captured through at least one camera of the AR device while being worn on the user's head.
Including, and
Comparing the non-linear and multi-directional virtual line representing the user's eye movement pattern virtually drawn by the user based on the tracked non-linear and multi-directional eye movement with the predetermined eye signature movement pattern. When,
Determining that the non-linear and multi-directional virtual line corresponds to the predetermined eye signature movement pattern based on the non-linear and multi-directional virtual line within a predetermined threshold of the predetermined eye signature movement pattern. When,
The transmission of communications to initiate a transaction to purchase the object in response to determining that the non-linear and multi-directional virtual line corresponds to the predetermined eye signature movement pattern. The communication includes said identification of the user, the local processing module transmits the communication through at least one network to a remote server that authenticates the user for the transaction based on the determined identification. And that it can work to
An AR display system that can operate to do. 23. The AR display system of claim 23, further comprising the remote server, wherein the remote server can operate to transmit a set of data relating to the transaction to a computer of a financial institution. The remote server is further provided, and the remote server is configured to transmit the communication data for authenticating the user to the computer of the financial institution associated with the user through the network, and the computer of the financial institution. 23. The AR display system of claim 23, which is configured to make payments on behalf of the user, at least in part based on the authentication. 24. The AR display system of claim 24, wherein the computer of the financial institution is capable of transmitting the payment to one or more computers of one or more sellers indicated by the user. 23. The AR display system of claim 23, wherein the eye-related biometric data further comprises an iris pattern of the user's eye on the AR device. 23. The AR display system of claim 23, wherein the AR device further comprises a microphone, which is further configured to receive biometric data including a voice recording of the user. 23. The AR display system of claim 23, wherein the eye-related biometric data further comprises a retinal signature of the user's eye on the AR device. 23. The AR display system of claim 23, wherein the AR device is further configured to receive biometric data including properties associated with the user's skin. The AR display system according to claim 23, wherein the eye-related biometric data further includes a blink pattern of the user's eyes. 23. The AR display system of claim 23, wherein the AR display system is individually calibrated for the user. The AR device is worn on the user's head and is the user's eye size, the user's head size, and the distance between the user's eyes, the distance from the AR device and the user's eyes, and the said. The AR display system according to claim 32, which is calibrated for physical features including the curvature of the user's forehead. 23. The AR display system of claim 23, wherein the identification of the user is determined at least in part based on a known iris pattern. 23. The AR display system of claim 23, wherein the identification of the user is determined at least in part based on a known retinal pattern. The AR display system according to claim 23, wherein the transaction is a commercial transaction. The local processing module is configured to detect interruption events associated with the AR device, which devices at least partially translate new biometric data into the detected interruption events and the new biometric data. 23. The AR display system of claim 23, further configured to capture from the user through the at least one light source and the at least one camera to reauthenticate the user based on. 35. The AR display system of claim 35, wherein the interruption event comprises removing the AR device from the user's head. The AR display system according to claim 35, wherein the interruption event comprises a loss of connectivity of the AR device to the network. 23. The AR system of claim 23, wherein the AR device comprises an eye tracking system. The AR system according to claim 23, wherein the AR device includes a tactile device. 23. The AR system of claim 23, wherein the AR device comprises a sensor that measures physiological data about the user's eye. 23. The AR system of claim 23, wherein the local processing module is part of a belt pack worn on the waist of the user. The AR system according to claim 23, wherein the local processing module is a part of a housing of the AR device mounted on the head of the user. 23. The AR display system of claim 23, wherein the non-linear and multi-directional virtual line representing the user's eye movement pattern resulting from the user moving the user's gaze comprises a plurality of linear sections. The non-linear and multi-directional virtual line is based on the non-linear division of the non-linear and multi-directional virtual line within a predetermined threshold of the corresponding linear division of the predetermined eye signature movement pattern. The AR display system according to claim 45, which is determined to correspond to the above. The eye movement-based user identification protocol further comprises transforming the format of the non-linear and multi-directional virtual line data, wherein the transformed data is a plurality of discrete areas of the virtual grid by the user. 23. The AR display system of claim 23, showing that the non-linear and multi-directional imaginary lines were crossed by the non-linear and multi-directional eye movements to be drawn. 23. The AR display of claim 23, wherein during the display of the real-world object in the direction of the user's gaze, the instructions are displayed and the non-linear and multi-directional eye movement of the user is tracked. system.

Images