KR20170047195A - Systems and methods for discerning eye signals and continuous biometric identification - Google Patents

Abstract

Devices, systems and methods are provided for providing substantially continuous biometric identification (CBID) to an individual using eye signals in real time. The apparatus is comprised in a wearable computing device, wherein the device is based on iris recognition using one or more cameras directed towards one eye or two eyes and / or other physiological, anatomical and / or dynamic measurements The identification of the wearer of the device is performed. Verification of the device user identity can be used to enable or disable the display of security information. The identity verification may also be included in the information transmitted from the device to determine appropriate security measures by the remote processing units. The apparatus can be used in a wearable computing device that performs other functions, including field of view correction, head-mounted display, view of the environment using the scene camera (s), audio data recording via a microphone, and / Can be integrated.

Description

FIELD OF THE INVENTION [0001] The present invention relates to systems and methods for recognition of eye signals and persistent biometrics, and systems and methods for < RTI ID = 0.0 > continuous < / RTI > biometrics.

In general, the present invention relates to systems and methods for continuous biometric identification (CBID), eye-tracking, and real-time eye-signal control by an individual . The CBID utilizes technologies belonging to the field of biometric-based authentication, encryption, and cybersecurity, and is implemented within the unobtrusive processor (s), where the inconspicuous processor (s) A head-mounted display, a remote display, eye tracking cameras, a scene camera showing the environment of the wearer of the device, and / or other wearable sensors. The CBID may be used in attempts to deliberately distort the identity of a device user using the iris code identification step within the System of System (SoS) architecture, the identification phase of other biometric features, and security interfaces with network devices, Security problems related to identity theft can be solved.

The widespread use of the Internet and computing / communication devices has led to explosive growth in electronic dissemination of information. However, verifiable control over the recipient (s) of the security information presents a significant problem in the cyber security field. Also, recipients of information can also be sources of sensitive information, where real-time knowledge of the identity of such sources can be an important security issue. An example of such a situation is knowledge of the identity of an individual entering credit card information (or account information) during the process of performing an online purchase. Today's technologies that are commonly used to remotely identify recipients or sources of security information are very vulnerable to deception. In the United States, identity theft affects roughly 15 million individuals each year, with an estimated financial impact of $ 50 billion.

Computers or telecommunications devices that receive information from the Internet are typically identified by so-called IP (i.e., Internet Protocol) addresses and / or identification codes that are typically embedded within central processing units (CPUs) or firmware. Although the IP addresses and / or the embedded device identity may be used to identify the device that is receiving or transmitting information, the IP address does not identify the user (s) of the device in a verifiable manner.

Systems that attempt to associate a device with an individual user typically employ passwords, security questions, and / or history records (referred to by terms such as "tracker" or "cookie"). However, these schemes can be easily avoided, for example, once the user has "logged in" This can be a very significant issue if the device is lost or stolen, or if access to the device is obtained by someone other than the intended recipient of the information. In addition, the schemes exist to mask or hide the true identity and / or the IP and / or hardware embedded address of a particular machine.

The most common cyber security methods for verifying a user's identity employ passwords and / or security questions. Once a password is entered and / or security questions are answered, it is possible to change users, which disables the role of the security system. There are also numerous ways in which passwords may be employed to obtain fraudulently and / or to answer fraudulently to security questions. These fraudulent methods intercept keystrokes during password entry; Guess passwords based on factors such as family names, places, pets, or simple alphanumeric arrangement; Decode information contained in packets when packets are transmitted over the Internet (public, wireless transmissions are particularly vulnerable access points); Automated sequencing using a set of commonly used passwords; Acquiring passwords via embedded malware; Pretending to be a legitimate website that requires a password; And other forms of so-called phishing phishing.

Biometric schemes for user authentication are becoming increasingly popular as a machine-based method for uniquely identifying individuals. Biometric identification involves sensing physical, genetic, physiological, and / or behavioral attributes that are unique to an individual. Real-time, real-time biometric authentication requires fast, non-intrusive, and non-invasive techniques.

As an example of a biometric authentication technique, U.S. Patent No. 8,432,252 describes a device that visually scans a finger to identify an individual and enables or disables secure communication based on fingerprint recognition. A number of patents utilizing authentication techniques based on finger scans have been granted to the applicants of USP 8,432,252. Automatic facial recognition, speech recognition, and signature recognition are the basis of other biometric methods. However, in general these techniques do not provide substantially continuous user authentication in an unobtrusive manner and are vulnerable to relatively simple methods of disabling security features.

For example, in most cases, these biometric devices can positively identify an individual when scanning. However, subsequent to this, it is possible for a separate individual to receive the security information or to become the source of the security information. Even at the time of a scan, devices used to display and / or input security information are generally not directly connected to users for user authentication. For example, automatic face recognition may be performed while a separate individual is performing keyboard input. These relatively simple methods of neutralizing a user's true identity are particularly problematic when there is a potential benefit to an individual who has been biometrically authenticated to intentionally hide or transmit his or her identity.

One example of a situation where a recipient of information may want to deliberately camouflage a true identity is the remote management of online scholastic examinations. In this case, the final recipient of the test scores solves all security challenges, while a separate individual may solve the actual exam questions. During management of Graduate Record Examination (GRE), Graduate Management Admissions Test (GMAT), Medical College Admissions Test (MCAT), and other professional promotion tests, complex identity swapping Sophisticated identity swapping schemes are regularly reported. Security issues related to remote management of academic and other types of examinations are expected to become increasingly important because educational and other knowledge-based service providers are increasingly looking for MOOC, distance- learning, and assessment formats that are being used in the future.

Iris recognition is now considered one of the safest biometric technologies. The iris shows a fine structure, which is an epigenetic phenotypic feature, which grows as a random element during embryonic gestation. Thus, unlike DNA fingerprints, they have completely unique iris pigments and structures, even if they are genetically identical twins (approximately 1% of the population). Another evidence of the welfare nature of the iris is that although the right and left (genetically identical) eyes of humans appear to have similar structures and colors, the textural detail of the individual left and right eyes is clearly distinct .

Although the iris can be seen non-invasively, it is in a well-protected organ (i.e., the eye) and, unlike a fingerprint, is generally protected from damage and abrasion. Although there are several medical methods that can change the microstructure and pigment in the eye, iris tissue generally remains surprisingly stable (e.g., unlike face features) over time, usually over a period of decades.

John Daugman first developed the iris recognition algorithm when he was at Cambridge University. Most commercially-developed iris recognition systems currently in use utilize Dougman's algorithm (e.g., as disclosed in U.S. Pat. No. 5,291,560). Commercially available iris recognition systems (e.g., Iris ID Systems Inc., BI2 Technologies, IrisGuard Inc., Eyelock Corp.) typically utilize portable or pedestal mounted devices and provide a maximum distance of 10 cm It works at distances up to several meters.

Still further developed and applied a two-dimensional Gabor wavelet (i.e., a special case of short sampled Frie transform) coefficients based on iris images collected using real-time video conditions. Complex dot products are computed by transforming images based on Cartesian coordinates to polar coordinates and applying 2-D Gabor filters to small areas, Reflect. Lack of sensitivity to the overall light amplitude (i.e., image brightness, contrast, etc.) helps to ignore variations in video recording conditions.

The most significant bits of inner products from different regions are combined into a so-called iris code, hereinafter referred to as "irisCode ". Most commonly implemented iris codes use 2 bits from each region to produce a 2048 bit (i.e., 256 byte) value, which can identify a unique individual from millions of people. The algorithms for calculating iris codes are coded as both software and hardware embedded firmware in CPU-based devices.

Today, more than 60 million individuals in 170 countries are registered for identification based on iris patterns. The Government of India is currently recording iris scans and fingerprints of the entire population of more than one billion people. Companies and government organizations that are leveraging these technologies are IBM, Panasonic, LG, Sarnoff, London Heathrow (Birmingham Airport, Gatwick Airport, Manchester Airport), IrisAccess (Korea), IrisPass (Japan) , The CanPass (Canada Nexus System), the Afghan repatriation program, and the United States Department of Defense detainee population management program.

The iris has a well-defined geometric shape that changes only by contraction of two opposing muscles (sphincter protrusions and expansive projections), and the two muscles control the diameter of the pupil. Thus, such uniformity and stability can be achieved by using conventional image quality and by providing an unprecedentedly low error match rate of 10 ^9.6 (i.e., one quarter billion), including contrast to head tilt and motion (false match rate).

Although the best understanding of the present invention will be obtained by reading this specification, this summary is intended to enable the reader to understand some novel and useful features of the systems and methods according to the present invention. Of course, this Summary is not intended to be a complete description of all features of the systems and methods herein, but is in no way intended to limit the scope of the claims set forth at the end of the specification of the present application It is not.

According to one embodiment, devices, systems, and methods are provided for a head mounted device that includes at least one processor coupled to at least one imager, wherein one or more imagers are Wherein the processor is configured to substantially, continuously, simultaneously, and / or periodically determine an eye-gaze evaluation, the imager detecting one or more characteristic features of the eye; Determining biometric data of a user including facial features of the user, voice or iris data, wherein the biometric data includes at least a user identification for access and control of at least a head mounted device, a connected device, a wireless device and a remote server And authentication.

According to another embodiment, devices, systems and methods are provided for a device user's substantially persistent biometric authentication (CBID). The device may be mounted substantially inconspicuously on the headgear, and the device may be embedded within or attached to conventional eyeglass frames or may be mounted within wearable computing devices such as those known as so-called Google Glass (registered trademark of Google) . More specifically, the illustrative embodiment includes one or more illumination sources (if desired) capable of illuminating one or more irises of the wearer of the device, one or more micro-cameras directed directly to view one or more eyeballs of the wearer of the device, A transmission mechanism or a reflection system, and a processing unit, the processing unit comprising: 1) analyzing images of the eye to determine the identity of the wearer of the device; And / or 2) sends the images of the eye to another processing unit that determines the identity of the wearer of the device.

In this specification, spontaneous eye movements intended to interact with a computing device are referred to as "eye signals ".

In addition, all information related to the biometric authentication information (e.g., iris code) and CBID must be transmitted with security, and the communication steps may include wireless communications with the headset device. CBID-based communications are securely performed in a manner that was filed on May 9, 2015 and is disclosed in a US application (Application No. 14 / 708,229) entitled " Systems and Methods for Using Eye Signals with Secure Mobile Communications & .

According to a first embodiment, devices, systems and methods are provided for performing iris recognition using a wearable device substantially, continuously, periodically, and / or upon request.

According to another embodiment, devices, systems and methods are provided that can establish the true identity of a user wearing a head mounted device.

According to another embodiment, devices, systems and methods are provided for establishing a true identity of a user through dynamic biometric measurements.

According to another embodiment, devices, systems and methods are provided, which provide users with easy, simple, and / or intuitive methods and systems for security that do not require a token, A password, and a physical device, and the physical device includes a credit card, a key fob, or other physical token.

According to another embodiment, devices, systems and methods are provided that allow ocular signals to be performed only when there is a verified biometric of the wearer of the device.

According to another embodiment, devices, systems and methods are provided that are capable of preventing user identity fraud and identity theft.

According to another embodiment, devices, systems and methods are provided that establish multiple levels of device security for at least one of a single user, a plurality of users, and unlimited access by all users of the wearable device Lt; / RTI >

According to another embodiment, devices, systems and methods are provided that use a head mounted device to authenticate users for at least one of educational, legal, licensing, and delivery purposes of services.

According to another embodiment, devices, systems and methods are provided, which include a head mounted device to authenticate users to allow access to at least one of a security room, a restricted area, a car, an airplane, Use

According to another embodiment, devices, systems and methods are provided, wherein images of both real and virtual objects viewed and recognized by a device wearer are stored in a personal augmented memory (PAM) ) Form. ≪ / RTI >

According to another embodiment, devices, systems and methods are provided that use a head mounted device to verifiably annotate the source of the document or data.

According to another embodiment, devices, systems and methods are provided that use head-mounted devices to annotate documents or data so that they can be verified or handled by an identified individual.

According to another embodiment, devices, systems and methods are provided that use a head-mounted device to annotate an actual or virtual object to be verifiably verified or handled by an identified individual.

According to another embodiment, devices using head-mounted devices to annotate such that wiriting, signature, or other handwritten information can be verified to be generated, modified, or reviewed by the identified individual, And methods are provided.

According to another embodiment, devices, systems and methods are provided that use a head-mounted device to be annotated so that the identified individual has spoken or verified that he / she has listened to the audible information.

According to another embodiment, devices that use head-mounted devices to annotate to be verifiable that an identified person has performed an act of including an action on an object and including some physical actions perceptible by the head- Systems, and methods are provided.

According to another embodiment, the behaviors, behaviors, and performances of identified individuals who are imprisoned, parole, probation, restraining order, or acted upon by a court , Or devices, systems and methods that use head-mounted devices for monitoring, limiting, controlling, or impacting biometrics to be verifiable.

According to another embodiment, devices, systems and methods using head-mounted devices for the purpose of verifiably assessing the poisoning status of suspect individuals affected by alcohol, drugs, or other substances at the time or for a period of time Are provided.

According to another embodiment, a device using a head mounted device for verifiably annotating a biometric response or other measurements of an identified individual participating in a supervised medical act for testing, treatment, or study of a drug or other prescription Systems, and methods are provided.

According to another embodiment, devices, systems and methods are provided for using a head mounted device for authentication of a purchase, wherein the authorized purchase is for on-line purchase security and off- Where the off-line includes a retail establishment or any location where the object is desired to be purchased.

According to another embodiment, devices, systems and methods are provided that use a head mounted device such that no other person than the identified individual is allowed to view the data.

According to another embodiment, devices, systems and methods are provided that use a head mounted device comprising a second imager coupled to an externally directed processor, the second imager being decoded by a processor And the code is one of a bar code and a QR code, and the processor decodes data representing information about the product.

According to another embodiment, devices, systems and methods are provided which use a head mounted device comprising a second imager connected to an externally oriented processor, the second imager utilizing image recognition To detect an object that can be identified by the processor, and the processor presents information about the product.

According to another embodiment, devices, systems and methods are provided, which use information about the product to allow an authenticated user to securely purchase the product.

According to another embodiment, devices, systems and methods are provided, wherein information in the area of the monitor actually seen by the identified device wearer is displayed (in real time) at a high spatial resolution or at high temporal frequencies frequencies) may be updated, while peripheral areas (i.e. areas not visible by the device wearer) may be displayed at a lower spatial resolution and updated at lower frequencies.

According to another embodiment, devices, systems and methods are provided, wherein information in the area of the monitor actually seen by the identified device wearer is displayed (in real time) at a higher spatial resolution and / When updated with higher temporal frequencies than the surrounding regions (i.e., not seen by the device wearer) that are displayed at a higher resolution and / or updated at a lower frequency, certain regions or regions within lower resolution regions, Or objects may be displayed with a higher spatial resolution than the lower spatial resolution regions and / or updated with higher frequencies. For example, facial features such as the eye can be displayed at a higher resolution when rendered outside the foveal area during high-resolution foveated rendering.

According to another embodiment, devices, systems and methods are provided for enabling one entity to initiate a secure communication channel with another entity by interrogation, wherein the security of the communication channel can be established prior to communication, It can be re-verified continuously or intermittently during communication.

In accordance with another embodiment, devices, systems and methods are provided that enable security protocols for coordination between parties to cause an action to occur, such that each party is able to determine the time at which their identity is being continuously verified with the CBID Lt; / RTI >

According to another embodiment, devices, systems and methods are provided for alerting an individual to the desire of one or more identified individuals to exchange selected information based on a cross-examination.

According to another embodiment, devices, systems and methods are provided, which enable security measures to maintain the privacy of the scene data and / or the temporal data associated with the audio data.

Aspects and applications of the devices, systems and methods provided herein are described in the following drawings and detailed description of exemplary embodiments.

A more complete understanding of the present invention may be obtained by reference to the following detailed description of the invention when considered in conjunction with the following exemplary drawings. In the drawings, like reference numbers indicate like elements or acts. Exemplary embodiments are now described with reference to the drawings.
1 is a diagram of the basic elements of an exemplary embodiment of a system for identifying a wearer of a device.
Figure 2 shows exemplary steps used to determine the identity of the wearer of the device.
Figure 3 is a flow chart illustrating exemplary logic used to substantially continuously derive a device wearer identification status.
4 is a diagram of one embodiment of a circuit for controlling viewing and generating security information based on a device wearer identity.
5 is a perspective view illustrating an example of locations of multiple illumination sources and multiple camera pointing in a single iris.
Figure 6 illustrates one embodiment of a multi-camera imaging system.
Figure 7 shows an example of headwear communication with a network.
8 is a flowchart of exemplary steps for performing an on-line, secure purchase.
Figures 9A and 9B illustrate a head mounted display wherein Figure 9B illustrates a sight shield, which prevents anyone but the wearer of the device from viewing the contents of the display.

In the following detailed description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of various aspects of the exemplary embodiments. However, those skilled in the art will appreciate that the devices, systems, and methods may be practiced without these specific details. In addition, other embodiments may be utilized and structural and functional changes may be made without departing from the scope of the devices, systems, and methods herein. In other instances, well-known structures and devices are shown or discussed more generally in order to avoid obscuring the exemplary embodiments. In many instances, the operational description is sufficient to enable those skilled in the art to implement various aspects, particularly when the operation is implemented in software. It should be noted that there are numerous different and alternative configurations, devices, and techniques to which the disclosed embodiments may be applied. The full scope of embodiments is not limited to the examples described below.

In the following examples of illustrated embodiments, reference is made to the accompanying drawings, which form a part hereof, and in which is shown by way of example various embodiments.

Iris Recognition System

Today, most iris recognition systems operate at distances from 10 cm to 20 cm between the iris and the camera (studies are under way aiming for distances of up to 3 to 10 meters). The system of the present invention is capable of substantially continuously performing iris recognition at relatively short distances of approximately 20 to 30 millimeters (20-30 mm) associated with unobtrusive headwear.

For these short distances, a typical image resolution may equal or exceed the image quality recorded at the distances used by current commercial iris recognition devices if adequate illumination is available. However, spatial aberrations, including substantial skew, can be generated by micro-lenses and / or other optics attached to iris-tracking micro-cameras. Spatial aberration can distort images of other structures within the iris and eye region.

The spatial aberration can be processed within the iris recognition algorithm by using a short focal distance platform that computes both the device-user authenticated iris code and the " target "(i.e., have. This maintains consistency between iris codes when they are compared.

Iris recognition also depends on the ability to acquire on-axis images of iris (i.e., images that are viewed perpendicular to the eye surface and that focus on iris and / or pupil) . When viewed at various angles (i.e., off-axis), the features of the iris can be distorted and thus cause loss of information. This can be especially problematic if the camera is unobtrusive and both are placed very close to the eyeball in the headwear. During normal movement of the eye, the iris may move off-axis with respect to the viewing angle of the unobtrusive camera, which may result in the loss of the ability to generate iris codes, The codes can be compared with the target iris code on a very strict basis.

In exemplary embodiments for handling this situation, the system can be used for high-stringency iris recognition when iris can be seen in on-axis mode combined with less rigorous iris recognition And / or feature recognition of other structures of the eye within the region if the iris is pointed off-axis with respect to the direction viewed by the camera.

The features tracked during off-axis periods are described in more detail below, but may include shapes of sclera, patterns and locations of blood vessels in the sclera, shapes of eyelids, and / or positions of eyelashes. Unlike iris, these features can change over a lifetime. However, these features do not change significantly over the time course of one session of wearing the device.

Thus, during normal operation, the user can directly face the camera to wear the device initially and then at least periodically register the high-rigidity iris code thereafter. Depending on the desired security level, accuracy and Gabor coefficients as well as differences or measurements of the Hamming distance (i.e., the measurement of the difference between the two vectors 0s and 1s, which are the same distance), can be adjusted to a wide range of severities . At times between registrations with high-strict iris codes, features within and around the eye region are tracked to ensure a substantially continuous user identity. Similarly, the threshold for the degree of match in the images of the recorded features around the eye, compared with the images when the identity of the wearer of the device is known, can be adjusted according to the desired security level.

For example, in comparison with commercially available systems used to identify (or expel) individuals when entering the country, the system of the present invention can solve the following questions: At any given time, Of the iris of an individual wearing the same iris is matched with a representation of the iris of one (i.e., target) individual iris? Technically, this can be much less difficult than proving that the iris image matches or does not provide a match in the database with millions of iris patterns.

The CBID system may be implemented in headwear, which includes one or more scene cameras that show the wearer's environment of the head-mounted display, eye-tracking, and / or device. Scene cameras may include sensors for capturing "scene" information, such as energy at the RF or IR wavelength, otherwise the scene information would not be recognizable by the wearer. The headwear may also include other sensors such as a microphone, an accelerometer, a touchpad, an ambient light detector, a galvanic skin sensor, a thermometer, a pulse oximeter, an electroencephalograph (EEG) (EMG), electrocardiography (EKG), heart rate variability (HRV) sensing local headwear positioning system, global positioning system (GPS), and / Electronic components that measure structural integrity and / or performance of a headwear device. Data that is substantially continuously acquired from these wearable sensors can be tagged with user identification codes to ensure positive identification of the information source.

Continuous Biometric Identification (CBID)

Figure 1 shows an exemplary embodiment of the elements of the CBID system 100 which includes a camera 125 oriented to view iris illumination sources 130,135 and 140, iris 115 and / or other features of the eyeball. ), And a processing unit 165. Anatomically, the iris 115 surrounds the pupil 145 and is itself surrounded by a sclera 150 or a white area of the eye including blood vessels and other identifiable markers. Periodically, the eyelids 105, 110 interfere with the field of view of the iris 115 and the sclera 150 during blinking or during spontaneous blinding.

The processing unit 165 may generate images of iris codes and / or eyeballs and may be coupled to a remote processing unit 160 that is (and is functionally connected to) a source and / or receiver of security information via a communication link 155 Lt; / RTI > Transmissions over the communication link 155 between the processing unit 165 and the remote processing unit 160 mounted on the headwear device 600 (FIG. 5) may be encrypted and transmitted over the wire and wireless transmission methods 155 ) Can be employed. Depending on the presence or absence of a substantially continuous positive identification, the remote processing unit 160 may perform steps to allow or hide the processing and / or transmission of the security information.

The illumination sources 130, 135, 140 may be infrared or near infrared LEDs or organic LEDs (OLEDs). The intensity of the illumination sources 130, 135, 140 may be controlled based on the target brightness levels sensed within the field of view of the camera 125, as described in US 8,890,946.

The micro-camera 125 may include sensing of electromagnetic radiation based on charge coupled devices (CCD) or complementary metal oxide semiconductor (CMOS) techniques, hereinafter referred to as an imager. The micro-lens 120 focuses the electromagnetic radiation from the region of the eyeball onto the sensing area of the camera 125.

The processing unit 165 may be a field-programmable gate array (FPGA), a microcomputer, a microcontroller, an application specific integrated circuit (ASIC), or other computing device. The processing unit 165 may be a single device or the processing functions may be performed by an assembly of discrete physical devices.

Figure 2 shows an exemplary embodiment of steps for identifying a device user. The image 200 comprising the area of the iris 115 is digitized using so-called "frame grabbing" techniques known in the art (process 205). By identifying the center of the mathematical expressions on the basis of the position of the largest black region in the digitized image 200 and / or the boundary 250 between the pupil 145 and the iris 115, The center is determined (process 210). Based on the transition between the black region of the pupil 145 and the less black, marbled region of the iris, the inner edge 250 of the iris may be determined (process 215). The iris 115 has an anatomical knowledge of roughly circular shape and is based on a transition between the marble color of the iris 115 or between the structured area and the white area of the sclera 150, Limbus limb) 245 is determined (process 215). In general, the inner edge 250 and outer edge 245 of the iris 115, when viewed from angles that are not perpendicular to the surface of the eye at the center of the pupil 145, . ≪ / RTI > Thus, the general shapes traced for the inner and outer edges 250 and 245 of the iris 115 are approximately elliptical. The detailed configuration of the iris codes should resolve typical deviations from the circular shape of the iris 115 (particularly, the boundary between the iris and the sclera 245).

Next, a Gabor transformation calculation (process 220) is applied to the pixels in the image 200 located between the inner edge 250 and the outer edge 245 of the iris 115. Next, the coefficients of the Gabor transform are assembled into a so-called "iris code" (process 225), where the most commonly used 2048- Assembled from the most significant bits. The area of the iris 115 may be ambiguous due to partial eyelid closure 105, 110 closure, shadow 240 or eyelashes 235. In some cases, In order to resolve such missing or unclear information, strict criteria may be associated with the iris code (process 230).

If the identity of the device wearer is determined via the iris code, the image of the area surrounding the iris 115 is stored and registered as belonging to the device wearer. The image may be subsequently used as a comparison criterion under conditions in which the iris code can no longer be calculated according to the eyeball turning. The image of the iris 115 is blocked because of blockage due to shadows 240, eyelids 105, 110 or eyelashes 235 or because the iris 115 has deviated sufficiently off the axis relative to the central field of view of the camera 125 , Can be ambiguous. When these conditions occur, the distinguishing features in the reference image can be used to substantially continuously determine the identity of the wearer of the device. These features may include supplemental biometric measurements (i.e., physiology, physiological, and biochemical) as well as the location and shape of the sclera 150, the locations and patterns of blood vessels in the sclera 150, the shape of the eyelids 105 and 110, And anatomical measurements), which will be described below.

3 is a flow chart illustrating a substantially continuous decision process for determining whether a wearer of the device is positively identified at a predetermined time. This determination process is initiated (process 300) by acquiring an image from the iris-tracking camera 125 and attempts to calculate the iris code. Methods for digitizing camera images and calculation of iris codes are described in more detail in FIG. 2 and are known in the art, for example, as disclosed in US 5,291,560. The next step (process 305) is to determine if the viable iris code has been determined. The reasons for the non-viable iris code are partially wrapped eyelids, fully wrapped eyelids (i.e., spontaneous blinding of eyes or involuntary blinking of eyes), shadow (240) obscuring iris 115, An eyelash that obscures areas of rod-like shape, and / or iris 115 that is rotated sufficiently away from the center of the field of view of camera 125.

If an executable iris code can be computed, it is compared to the predetermined iris code of the intended or target device user (process 310). The severity of this comparison can be adjusted according to the desired level of device user security. If there is a positive match with the wearer's identity, the time of the positive match is set to the current time obtained from the free-running clock 335 (process 315) The features in the surrounding area are stored and registered as the target device user (process 320) and the Boolean indicator of the positive identification status is set to "true" (process 345).

If the iris code can not be calculated (process 305), a check is performed (process 325) to determine if the iris code was unavailable for an extended period of time. If the iris code has been recently determined, the features in the image of the area around the eye are compared with the stored images at times of positive identification (process 345). If there is a sufficient match of these features, the Boolean indicator of positive identification remains "true" and a new image is acquired.

If the newly identified iris code does not match that of the intended wearer of the device (process 325), or if the iris code is not calculated for a predetermined time (process 330) and there are insufficient matches of features around the eye 340); The difference between the current time registered by the pre-running clock (process 330) and the most recent time of positive identification is calculated. If it is determined that this difference value is greater than the predetermined threshold time, the Boolean indicator of the positive identification state is set to "false" (process 340).

The predetermined threshold time is adjusted to a value that allows the inability of short periods, for example, to positively identify the user during a "normal" eye flicker of the wearer of the device. Thus, the determinations "continuously" and "substantially continuous ", as used herein, may include the inability of these short periods to positively identify the user. In general, the threshold time is set to a value between 0.1 seconds (i.e., the duration of a fast blink) to 10 seconds, where the latter value is minimum security, So that you can see them while you are there. A typical threshold is 0.3 seconds,

Once the new image is acquired (process 300), the overall decision process is repeated to provide a substantially continuous identification. The user identification state may be attached to all transmissions related to the system (process 345). Alternatively, the images of the iris 115 may be transmitted to the external processing unit and the decision process for determining whether the device wearer is positively identified may be performed remotely.

4 is an exemplary embodiment during applications that control the transmission of security information 155 via display 710 and / or external devices. Images of features in the region of the iris and / or eye region are substantially continuously acquired by the camera 125 and based on the degree of match in these images, a substantially continuous device wearer identification state is detected by the processing unit 165 ). The device wearer identification state is transmitted via the communication link 155 to the external processing unit. Thereafter, any number of other processing units 400 may cause the security information to be viewed and viewed by the substantially continuously verified individual.

The biometric authentication may occur during one or more times (i.e., sessions), or may occur in any combination of times before, during, or after the eye-signal control process requiring user verification. For example, an individual who is taking a test may be required to perform an ocular signal command sequence that causes biometric authentication, both before and after the test period. Online purchases may require authentication not only simultaneously with the eye-to-signal sequence causing the transaction, but also before the transaction is performed (e.g., during a pre-authentication phase for the total cost of the transaction). Reading the security document using eye-signal controls, and possibly other controls, may require repeated biometric authentication, which is synchronized to all of the eye signals used during the reading process.

Supplementary Biometric Measures

Yet another embodiment includes the use of anatomical and physiological features to substantially continuously and continuously monitor both an individual's identity and a functional state. For accurate tracking of ocular signals, the shapes, dimensions, and / or relative spatial positions of the eye's anatomical features must be known to the headwear device. Thus, during normal operation, these features are generally available to devices, since they form the foundations for many of the adjustment factors used to generate accurate eye tracking. In addition, they can also be used to verify (substantially continuously) the identity of the wearer of the device. This can be achieved, for example, during iris flickering, when the iris is ambiguous by eyelashes or other structures, and / or when the iris is pointing away from the camera (s) looking at one or two eyes It is particularly useful during periods of inactivity that are not continuously available. Examples of such anatomical features that may be components of the user identity include:

Corneal radius (including second-order parameters because the shape may be ellipsoidal)

Pupil depth

Eyeball diameter

● limbus radius

● Offset between optical and visual axes

• Synthetic lens parameters (cataract surgery to replace lenses with artificial lenses is performed 2.7 million times a year in the US).

● pinguecula (a macular thickening of the conjunctiva of the eye's sclera), pterygium (wedge-shaped growth), strabismus (crossed eyes), amblyopia (lazy eye ) And various diseases

● Normal eyelid position

● Eyelash thickness, density, color, range, length

● eye fissure shape and measurements

● Eyebrow according to the field of view of the camera hair, structure, position, color,

● Skin composition and surrounding tissues of the eyelids according to the camera's field of view

There are other anatomical factors, such as the size and shape of the nose (which is generally influenced by how the device lies on the head, thus affecting the positions of the cameras and illumination sources relative to the eye) It is less useful for user identification purposes.

However, the measurements of the distances between the cameras and the eye features (again, largely influenced by how the device lies on the nose), for example, determines whether the device has been removed from the head Can be used. If the device has been removed from the head, then the device is put back on the head and the device performs high security behaviors and transactions until a high-rigidity (e.g., matched iris code) authentication procedure is performed A prohibiting indicator can be set.

In addition, the physiological responses measured by the device may be components of unique identifiers in an individual's identification or "signature ". Examples of physiological features that are measured by a device and thus can be components of a user identity include:

• The degree of contraction or expansion of the pupil in response to changes in light (which is highly dependent on the age of the wearer of the device) or a specific stimulus (light, emotional response, cognitive load)

The pupil contraction or expansion rate in response to changes in light (rate)

Horizontal and vertical displacements of the pupil as the pupil contracts or expands

● Scope and speed of voluntary saccadic movements

The range, frequency, and speed of micro-saccades

The degree, presence, extent, frequency, and pattern of tremors, drifts, and other eye movements

The degree of correlation between vergence eye movements (when two eyes are monitored)

● The speed at which device wearers identify and track new targets

● Flicker behavior such as frequency, speed, duration,

● How the user performs eye signals, which may be part of the grammar or interaction model for eye-control

Eye signals in terms of visual stimulus, biometric data, and other data including location, time, activity, physical or cognitive state and intent,

Maybe based on the movements of the eyelids and surrounding skin tissue, the skin, muscles and movements that are cataloged in the Facial Actions Coding System (FACS).

● Various parameters related to the "experience" of the wearer of the device (compared to novice users)

The back end features of the list, the device wearer experience, may not be strictly classified as "physiological" measurements. The experience is reflected in the cluster of coefficients maintained by the device, which includes the time consumed with the device, fixation times, the history of the reaction times, the contextualization of the various data types, Reflects the user ' s ' experience ' mainly based on the frequency of unintended selections (i.e., corrected later). One example of such data used by the device is the so-called "eyemovers" (i.e., make a 1 of N selection) The targets which urge the eye (s) to move).

The user experience is an example of a number of dynamic features that can be measured when performing actions such as measuring an object or reading a character. These "behavioral biometrics" or "styles" that perform actions can be based on factors such as past experience, familiarity, interest, and so on. Some hyperactive features may include anatomical and physiological influences. As a parable, the gait of an individual is a large-scale example of such an act. In the case of ocular signals, the exact attachment points and the contraction force of the six muscles controlling each eye can affect behavioral biometrics of eye movements.

As described in Systems and Methods for Biomechanically-Based Eye Signals for Interacting with Real and Virtual Objects for interacting with real and virtual objects, One of the ways to classify neurons is the application of neural networks trained to recognize the characteristics of an individual's eye movement. User experience as well as specific motor characteristics can be used to identify an individual (ie, an "eye-gait" of an individual, by analogy).

For example, in some cases, such as identifying identical twins, unique variations of some anatomical features may be too small to be actually used for biometric authentication, but these anatomical, physiological, and / .

Given these shortcomings, this is a combination of independent features that provide differentiation power.

If the measurement of the diameter of the limbus is a resolution that can consistently (i. E., Iteratively) classify individuals into one of ten groups and that is obtained during large saccadic movements If the maximum velocity (assuming that it is not dependent on the corneal limbus diameter) can further categorize individuals into one of 15 groups, in identifying an individual, the combined power of these two independent measurements is 150 (10 * 15). The multiplicative nature of the independent and distinct features provides considerable power to distinguish individuals.

In particular, anatomical, physiological, and dynamic parameters maintain the persistence of user authentication during periods when iris authentication is not available (e.g., when the iris is obscured by the eyelids, eyelashes, or out of focus) Can be used. In addition, in many applications, the device is required to determine whether the wearer's features match one, identified person's characteristics, or a small number of known individuals. This is a much simpler sorting process than, for example, identifying whether one of a large number of individuals can be excluded from wearing a device. A deviation in one or more anatomical, physiological, or dynamic measurements may indicate that the device user has changed.

These identification features may also include other security tokens, such as information tokens (passwords), physical tokens (keys), generated tokens (speech, gestures, ), And the like.

Another exemplary embodiment of the systems and methods herein includes using iris recognition to find and retrieve calibration parameters associated with an individual user. It is common for unknown or new device users to know the calibration parameters when they wear the device on his / her head. An initial iris code may be generated through a process that directs the user to gaze in the general direction of the camera looking at the eye, or through a process in which one or more cameras look at the iris during normal eye movements.

Multiple Camera Configurations

Iris images seen from different directions using portions of iris images taken at different times by one camera and / or multiple cameras when the iris portions may be ambiguous during the normal movement of the eye are referred to as " Stitch together "may be useful. The use of multiple cameras to view the eyes is illustrated in Figures 5 and 6.

Alternatively, the iris codes themselves may be "stitched together " from different images of the iris (i.e., computed after the segmentation process). Again, these images may be taken at different times, or taken by different cameras or at different times and combinations of camera angles. As an element of this approach, portions of the iris that are successfully seen in a plurality of images can be averaged (before or after calculating the iris code), which improves reliability and / To prevent the effects of image noise.

One of the problems associated with unobtrusive CBID is the fact that the camera (s) that are aimed at the eye are generally off-axis relative to either the visual or optical axis of the eye . This causes an "off-axis" view of the iris. Also, since the eye moves during normal eye-to-eye functions, the degree to which the iris is "off-axis" changes. If the iris template is registered while the user is looking in a certain direction, it will become increasingly difficult to identify the same iris because the eyes rotate from the direction in which the iris template was registered.

One solution to the "off-axis" is to use multiple cameras that observe the eye at different viewing angles. At any given time, images may be selected from the camera closest to the on-axis, such that the on-axis may be selected, for example, at the inner and outer iris boundaries (relative to being very oval) Can be grasped. The camera that is most "on-axis" may change because the eye moves during normal operations.

Another solution to this problem is to store multiple iris templates for an individual, collected at different viewing angles.

As a result of eye tracking, since the user's view direction is known, it is possible to: 1) store iris codes in a plurality of known viewing directions, and 2) thereafter, collect iris codes collected at any time in known viewing directions in the same or nearest viewing directions, Compare with iris codes.

The degree of off-axis viewing can be reduced using multiple cameras and / or multiple iris codes. By selecting images from the camera closest to the on-axis, a most direct view of the iris can be obtained. Both registering iris codes from known individuals and registering the identity of an individual in real time can be based on images of the most on-axis camera, which is to improve identification robustness.

Similarly, all of these schemes are consistent with numerically "rotating" the image of the iris to be on-axis before registering the template or calculating the iris code in real time for identification. In many eye-tracking systems, the angle between the optical axis and the direction of the viewing camera must be known in order to calculate the gaze direction. This provides key coefficients for applying a rotational transformation, which allows the image of the eye to be seen as seen along an axis perpendicular to the center of the pupil (or corneal limbus).

These iris codes may be collected using conventional lenses or holographic lenses to accommodate the desired views. Also, iris codes may be collected, wherein the images of the eye are reflected at one or more of the reflective surfaces. These surfaces can be designed around a single, large reflective surface or multiple reflective surfaces (i.e., multiple micro-mirrors). These mirror-based systems may be of the so-called free-form optical pathway, which is in the form of conventional reflective surfaces or designed to minimize the physical characteristics (e.g., weight, size) Lt; / RTI > They can also reflect light of selected wavelengths (e.g., a so-called "hot mirror") to enable CBID without disturbing views in the light of visible wavelengths. The free-form optics design allows cameras to be positioned unobtrusively along the edge of the eyewear, while maintaining the ability to see around the optical axis or optical axis. The images of the eye (s) may alternatively be collected by one or more detectors (e.g., photodiodes), wherein the spatial selection and resolution is based on switchable Bragg gratings-based: SBG Quot;) < / RTI > devices.

5 illustrates an exemplary system 600 that illustrates placement locations for a plurality of illumination sources 130, 135, 140 and a plurality of cameras 125. As shown in FIG. Both of these illumination sources 130,135 and 140 and the cameras 125 are directed towards an area of one eye that includes the iris 115, the central pupil 145 and the surrounding sclera 150. The use of a plurality of illumination sources 130,135 and 140 in comparison to a single light source 400 can be achieved by using iris 115 which is better illuminated for a wider range of eye movements by the wearer of the device And reduces the tendency to create shadows 240. The use of multiple cameras 125 improves the ability to capture on-axis iris for a wider range of viewing angles by the wearer of the device, and more acute angles ≪ / RTI > reduces distortion of iris images resulting from camera-based imaging.

The cameras may also be configured to capture the views of eyes, eye features, eyelids, eyelashes, eyebrows, surrounding skin, and facial muscles, non-contiguous vowels, or adjoining regions stitched together. The plurality of cameras may be supported by a plurality of LEDs or illumination sources configured to illuminate areas viewed by the cameras. Capture for eyes, pupils, features, eyelids, eyebrows, and skin can be used to measure emotional response to stimulation from the on-screen display or stimulation within the user's environment have. This can be used, for example, for interpersonal communication, for emotional reactions related to the gaze direction, and for emotion-based interactions in games. The association between the gaze directions of the players measured by the cameras can be combined with other facial information to be interpreted to define the emotional state of the user. Actors of a game or entertainment experience may be synchronized with the user's eyes, facial, head, hand, body, and biometric behavior, reacting with behavior and away from the user or with the direction of the gaze towards the user, This communicates emotions through rendered facial features, body, and eye-movement, and stimulates the volatile and dynamic levels of emotional expressions and interactions of the character / user. These actors may be artificial intelligence characters, or they may be avatars for remote human characters, and the user may also be represented by a remote avatar.

In the case of the system 600 illustrated in FIG. 5, the components are mounted within a pair of eyeglass frames. Fig. 5 shows only the left half of the glasses which are directed towards the left eye, which is for the sake of simplicity.

The eyeglass frames are worn on the head using a nose piece 510 and an ear stem 515. In this example, the on-board processing unit 165 is disposed in the left ear stem 515. Although not shown in this figure, a battery pack that supplies power to the device may be mounted, for example, to the right ear stem. Although not shown in this figure, the scene camera on the frame may optionally be used to view the environment of the wearer of the device. Optionally, one or more display-providing monocular or stereo imagery for the user's eye (s) may be mounted to the frame, although not shown in this figure. The frame provides an image that enhances the view of the user's real world, an image provided on the off-axis or not in the user's direct line of sight, or immersive imagery creating virtual reality Display (s).

An exemplary configuration of multiple cameras is illustrated in FIG. The plurality of cameras 2000a-f can avoid obstacles, including the upper eyelid 105, lower eyelid 110, and any occluding eyelashes 236, through multiple views of the eye. When viewed at different angles (i. E., When viewed by different cameras), the areas that are blocked by certain obstructions (e. G., Eyelashes 236 of Fig. 6) are different. Under these conditions, full representations of the surface of the eye can be reconstructed by extracting or stitching together information from the images collected using images from different cameras. Different cameras with different views of the eye may have overlapping or non-overlapping views. In addition, different cameras may have different lenses, optical filters, frame rates, or resolution. There may also be a mix of 2D or 3D image cameras. Any combination of camera or cameras 2000a-f may be available at any time (e.g., during power on and function). Access to the images collected under the range of sampling conditions improves iris recognition, pupil recognition, feature recognition, and robustness of eye-signal recognition.

In addition, the use of multiple cameras can be used to form three-dimensional views of the eye. An alternative method for viewing three-dimensional structures in the eye is spatial phased imaging. Regardless of the ways in which the shapes, sizes, and positions of structures in the eye are determined, their knowledge of the three-dimensional morphology can be used to determine the resolution, convenience (e.g., , And toughness can be increased.

Many-cameras (i.e., each eye viewed by multiple cameras) directed toward both eyes further increase the accuracy of both vergence measurements. Binocular conduction literally adds an additional dimension to eye-signal control. For example, by using binocular conduction, icons in different layers of select panels can be specified by the wearer of the device.

A number of cameras that simultaneously observe substantially at wide angle ranges can also be used to observe the movement of surrounding features, including eyelids, skin, eyebrows, and even portions of facial muscles. The movements in these features can be used to extract other user conditions including the emotional state. In an exemplary embodiment, other ocular measurements indicative of emotional states include pupil dilation, lens shape, and heart rate. In further embodiments, it is possible to determine the heart rate of a user by using one or more cameras to determine the ratio of small expansions and contractions in images of the vasculature of the eye, in particular of the vasculature within the sclera .

Determination of user conditions including emotional states

Additional sensors may be added to the headware to determine device user conditions and emotional states. Examples of substantially continuously monitored data include pulse oximetry, galvanic skin resistance, EEG, ECG, and temperature sensors.

The device knowledge of user emotions can be used for identity verification as well as for regulating a wide range of applications. Emotion can be a powerful adjunctive input, especially for gaming. For example, a game's response to fear or stress by a user may facilitate tasks associated with the gaming process. On the other hand, the stress expressed in a multi-user game may be used to trigger a defeat. Reflections of "real" human emotions can be connected to the avatar, where actions and responses of the avatar are influenced by emotional states.

Another area where knowledge of user emotion can be valuable is the field of conferencing, which involves the generation of synthetic facial expressions. Compared to transmitting video streaming during teleconferencing, creating and showing one or more composite faces can greatly reduce the bandwidth required between meeting locations. The problem that arises with regard to viewing composite faces is the absence of proper facial expressions. This tends to create "uncomfortable" feelings in some of the viewers. Knowledge of the emotional state (and true identity) of the individual, which is the source of the meeting content, allows for more appropriate and dynamic synthetic pyramid expressions to be generated. Emotional states may be limited or broadly broadcast to any number of recipients (or as desired) or to a subset of recipients.

In interpersonal interactions replayed through synthetic faces or avatars, it is also possible to separate interactions into a pair of individuals or small groups (even in the "virtual" presence of other avatars ). This is particularly useful in both gaming and teleconferencing applications. Similar to the actual conversation between individuals in a room, multiple simultaneous virtual conversations can occur, for example, without limitation to the identity or geographical locations of the actual participants, such as "virtual eye quot; -to-eye contact ", < / RTI >

Cognitive load is a major contributor to many emotional states. Under certain lighting conditions, the pupil diameter highly reflects the cognitive load. Thus, if the illumination conditions are known (the illumination conditions can be evaluated from the overall intensity level observed in the scene camera images), it is possible to evaluate the cognitive load substantially continuously by observing changes in the pupil diameter. Similar to the knowledge of the emotional state, including cognitive load as input may include game, conferencing, pace of document review, rate and / or level of difficulty of test questions, evaluation of ad effectiveness, Medical assessments including post-traumatic stress disorder, psychological assessments when viewing images, and so on.

Using one or more scene cameras in combination with multiple cameras for viewing the eyes (i.e., viewing the user's environment) can provide greater possibilities for an immersive environment. For example, in game operations, the wearer's surroundings may be projected within the ambient environment (s) of the avatars or within other representations of the players in the game space. Viewing head movements and / or objects in the "real world" can be translated into a virtual environment. The conversion of real and virtual environments is particularly effective for device wearers within a three-dimensional "cave" projection system, but similar effects can be obtained using large display screens or multiple display screens.

The creation of an "environmental context" using multiple scene cameras can provide increased user satisfaction, safety, performance enhancement, and so on. For example, it is possible to adjust the orientation of images during a meeting to view the places being viewed in real time by one or more selected conference participants. Knowing what an individual (especially a conference participant) is focusing on is a great help in interpreting what the person wants to convey. Conversely, if it is clear that one or more audience members do not appear to be interested in the content being served, then this is the clue to change the content.

CBID-based eye-signal device setup

Depending on the application (s), the functions of the CBID-based eye-signal device can be implemented in a number of settings. The processing and / or authentication may be self-contained and / or performed remotely. Authentication can be applied to the outgoing data and / or the flow of incoming data can be limited. Device operations may be limited to one person or group of people, or may be unlimited. The device operation may be further limited to perform tasks only under certain operating conditions. The display of information can be hidden by everyone except the user of the device, or it can be viewed by everyone. Authentication may be applied only if it is a user request, or it may be automatically applied to some or all data. The eye signal control of other devices may or may not be limited to an individual, a group of individuals.

The following table lists some of the classifications of the settings and their respective examples. Definitions and abbreviations used to describe eye signal device settings are as follows.

IC - iris code (irisCode): The result of applying pattern recognition techniques to images of the eye to quantify the welfare patterns in the iris into comparable bit - patterns for biometric authentication.

EIC - encrypted IC: an iris code encrypted so that it can not be back engineered with the original image of the iris or any other iris-based derived parameter

TEIC - Target EIC: The identified EIC, where the matching with the IC calculated from the image of the eye is relevant and therefore represents a friendly biometric.

CBID - Continuous biometric authentication: a biometric authentication iterative process, which can be performed on a headset device or remotely by sending EICs or images of one or both eyes to a remote processor. The CBID may occur at a fixed rate (e.g., 30 times per second) or asynchronous rate (e.g., only when the device is moved and re-mounted).

UUID - universally unique identifier: A firmware-encoded unique identifier (e.g., a numeric code) for any processing device containing headset devices.

GPS - Global positioning system: A satellite-based navigation system that can determine the location of a device anywhere near the Earth's surface.

Table 1: Classifications of CBID Device Settings Class Explanation case Stand-alone, 1 of 1 One TEIC is stored in the headset and off-headset communication is not required for CBID Otherwise approving the sole owner / user of the inoperative device Stand-alone, 1 of N "N" TEICs are stored in the headset and no off-headset communication is required for CBID Approving individual members of the family, all of the family are allowed to use the device (e.g., the calibration factors associated with each user are dynamically loaded Remote TEIC, 1 of 1 One TEIC is sent from the remote site to the device Pay-per-view rental of headsets Remote TEICs, 1 of 1 A number of "acceptable" TEICs are sent from the remote site to the device Clusters of universal headsets made available to some or all employees in the business Headset transmission EIC Continuously or periodically, the headset sends the EIC to the remote site where the user ID is determined (often from a large database) Online purchases from online "store" where users are registered without restriction on devices used for purchasing Headset transmission EIC and UUID Continuously or periodically, the headset sends the EIC and UUID to the remote site where the user and headset ID are identified (often from a large database) Online purchases from online "stores" where both users and specific devices are registered Excluded TEICs Using a CBID to determine if a user should be excluded from use or other access, where comparison is possible with a headset or remotely Determines if the user is on a no-fly list
Associating data with a single individual Search or create only the data file (s) to be decrypted under one person's CBID Secret list of traditional passwords By a single individual
"Eyes-only" viewing " CBID with HUD in place with field-of-view breaker; Data is transmitted only if there is a friendly CBID; Otherwise, dummy data may be sent. Take a test at a large online course Group information release The data file (s) that can be decrypted only under the CBID of one member of the group of individuals, Medical records available to primary care physicians, specialists and patients Release structured information Data content that may vary based on what is known about the identity of the viewer Structured ads based on the demographics of the identified viewer Data set displayed on the HUD Confirmation that a data set (e.g., text body) has been sent to a particular CBID user, the user can see any or all of the data set Device license agreement sent to user Data set shown on the HUD (E.g., text, pictures, graphics) along with the CBID along with the respective data set items to see if a specific user has actually watched it using eye tracking Ensure that all components of the legal document are shown A data set provided on a display device Confirmation of whether a data set (e.g., text) has been transmitted to a particular CBID user ' s display (e.g., monitor, phone, tablet)
The user can see any, all or part of the data set Confirmation of change notification of items and conditions sent to the user Data set shown on the display device A confirmation of whether a particular CBID user has actually watched the entire data set (e.g., text, pictures, graphics) using eye tracking directed at an external display device, the scene camera can confirm receipt of the content Using the QR code) Confirmation of notified consents related to legal documents Checklist Verification Using a scene camera combined with eye tracking and CBID, verified whether the identified individual saw each item in the checklist Pre-flight inspection by pilot "Black box" recorder CBID (and other data) stored in the non-volatile memory of the device for a predetermined period of time (several hours, days, etc.). The data set may include eye tracking and / or images (e.g., to determine what was seen prior to an emergency) Identify vehicle drivers and identify those that interfere with driver concentration prior to an accident User-Dependent Applications Based on the CBID, it provides access to a set of individually selected applications and data sets Search email based on user identification, and separate email and application access for other users User-tag data Allows the user to automatically add his / her identity to any set of transmitted data Outgoing text and email can be tagged to indicate that the CBID user is an author Electronic signature Digitally signed "series of simple steps including the CBID and timestamps the data set Electronically "sign" legal documents Mutual identification The data set (s) transmitted only in the presence of multiple CBID-certified individuals (applied to two or more individuals) Supervision of exams that must be held in the presence of both students and supervisors Mutual identification with explicit user authorization The data set (s) transmitted only in the presence of CBID-authentication and explicit authorization by all individuals, Exchange of personal information between people you just met An object in scene images Bar code, QR code or CBID-scene camera used to recognize objects / containers viewed from any direction seen by the identified individual Purchasing activity performed in a physical store (Brick and mortar store) that does not require a cash register or cash register A specific object at a known location EIC and GPS location sent to the processor for authentication Remote control of door opening for authorized individuals only Location-Sensitive User Identification EIC and GPS location sent to the processor for authentication Allow or limit access to buildings based on user identification Geographical limitation CBID combined with GPS to track an individual's location Track and allow access to certain resources by individuals under house arrest Interaction limited CBID < / RTI > combined with image identification within the scene camera images to identify the individual ' Tracking individuals who are not allowed to interact with the casino or other individuals Time-sensitive user identification Release information and / or user control based on CBID and current date / time Guaranteeing legitimate purchases of alcohol or other age-restricted items Other biometric sensors Data from other sensing devices associated with the CBID for long term monitoring and / or identification of pulse oximetry (e.g., irregular pulse), EEG (e.g., epilepsy), and / Automated 911 calls to user identities (which can be linked to hospital histories), "vitals" and geographic locations Unlock / access other devices
Allows on-device access to a library of access codes, tokens, passwords, etc. to interact with other processors that require some form of user verification if a friendly CBID is present Interacts with an automated teller machine based on CBID Highly secure applications A headset integrated with antispoof measures, including monitoring induced pupillary responses, heart rate (e.g., by visualizing blood vessels), headset tampering, To gain access to highly secure military sites Historical record CBIDs with time-stamps acquired with other data streams (video, audio, biometrics, GPS, etc.) for maintenance of history records Demonstrating past behavior of the individual under investigation Personal Privacy Automatically encrypt the acquisition of data sets to limit access to the actual device wearer at the time the recording was made (within the recording headset) Limit access to audio / video for private conversations Group Privacy Automatically restrict access to historically recorded data sets for any CBID-capable device wearer present at the time the recording was made Limit access to audio / video for meetings to participants Extended personal memory CBIDs with time-stamps obtained with other data streams (video, audio, biometrics, GPS, etc.) for personal use Find records of events related to other events With mutual identification
Extended Personal Memory Using encryption, viewing a historically recorded data set can be limited to participants identified in the interaction (especially when viewed at the time of recording). Restrict access to private conversation datasets to CBID participants Indexed extended personal memory Within the recorded data set, include searchable logs of CBID and other components in time-stamps, GPS locations, and data streams along with recognition of objects and / or words in audio / video data streams box The answer to the question: Where did I last see my car key / when? User-specific data retrieval Data feeds and searches can be tailored to CBID individual interests Hockey (better than basketball or other sports) score first Local device control


Wireless control of local devices (no internet required) Controls the home heater by the identified adult (ie, not the child)
Remote device control Remote device control using the Internet Remotely light the entrance of the home

While the above table refers to transmission and comparison of EICs, in some cases it is algorithmically possible to convert the images of the eyes to ICs and subsequently to EICs. Thus, the CBID may equivalently include comparisons and / or exchanges of information including images of iris, ICs, EICs, or other derived parameters. Similarly, the databases used for biometric comparisons are equally (for identification), ICs, EICs, images of eyes, images of faces (including eyes), images of irises, , Images of other features of the eye, including so-called "unfolded" (i.e., represented in polar coordinates), veins in the sclera, corneal limbus, or other captured or derived parameters . Thus, exchanging or comparing EICs also refers to exchanging or comparing any other derived data sets for biometric authentication.

Once the iris code is calculated, it can be used for searching in the database of known iris codes and / or supplementary biometric measurements. Once a match of the iris code (and other measurements) is found, the associated calibration factors and other information (user name, age, sex, etc.) are returned to the headset device for use during eye tracking and other actions Lt; / RTI > This search can be performed in a number of different platform configurations.

For example, a headset may itself include a plurality of (typically a small number of) iris codes and calibration sets, so that any member of the family can use the headset. The iris code generation, retrieval, and matching are performed entirely on the headset.

An iris code or iris image may be sent to a specific remote processor for identification and matching. The processor may serve a set of headwear devices, for example, at home, business, school, theater or geographical location.

• An iris code or iris image can be sent to the cloud for identification and matching. A cloud-based search for the identified person's data sets makes it possible for personalized calibration and other data to be used for any headset device that is used anywhere and at any time. The unaided retrieval of the correction and other data will appear to the user as an "instant on" feature of any wearable device. The overall data flow for this setting is shown in FIG.

Once user identification is established, it is also possible to apply the calibration set collected on one device form factor to another device form factor. For example, if the design of the device is upgraded and the position and / or orientation of the camera (s) and illumination source (s) in the headset are affected by the upgrade, then known displacements displacements may be added to the calibration sets. This allows the user to utilize a single set of corrections between multiple wearable devices that may have the same or different form factors.

In further embodiments, the CBID can be used to enable or disable "eye signals" or eye-signal controls as described in Systems and Methods for Biomechanically-based Eye Signals for Interacting with Real and Virtual Objects. The performance of a subset of all eye signals or eye signals may be made according to the presence of the CBID. These eye signals may include eye signals made only by one or two eyes of the wearer of the device as well as eye signals in the case of interaction with the real world or other virtual objects. For example, eye signals may include looking at a gesture made by a finger of the wearer of the device (using outwardly directed scene cameras) by the device user. These gestures may be exclusively valid under the CBID conditions (i. E., Result in behavior).

In yet another embodiment, CBID-enabled) actions may be limited to times when the identified individual (s) or other identified objects are within the field of view of one or more scene cameras . Face recognition, object recognition, and / or other identification types can be used to verify the co-location of these objects, including the wearer of the device and one or more other, identified individuals. Actions may be limited to being performed only in the presence of these individuals, for example.

Alternatively, the presence and / or approval of one or more other individuals may be performed by another CBID-based device worn by the individual (s). In this case, the individual (s) may be co-located with other device-identified individuals, or they may be authenticated (including intervening data transmission devices) directly transmitted between the devices Or remotely located with authentication sent via more centralized (e.g., cloud-based) authentication services. In the case of co-localized individuals, the transmission of authentication credentials may preferably be further limited to near field communication (NFC) Use is excluded).

In contrast, one or more identified individuals specifically designated to be absent during the performance of an action and / or the presence of an unauthenticated individual within the wearer ' s perimeter environment may be used by the device to perform any action or subset of possible actions Can be prohibited. This prohibition may be applied to any of a variety of wireless devices including certain headset devices (e.g., identified by a UUID during transmission), cameras, microphones, hostile appearance (e.g., in military or policing situations) Can be extended to exclude the presence of objects from the surrounding environment.

Similarly, the performance of an action and / or the presence of an object (including a person) in the environment of the wearer of the device may cause the capability to be temporarily or permanently abolished, and the ability to fragrant CBID- May be the cause of the abolition. Revocation can also be performed based on commands from a remote source, which can determine, for example, whether fraudulent activities are being performed or are being performed. Revocation of authentication credentials may be limited to individuals, groups of individuals, devices, certain functions or groups of functions.

CBID-Enabled, Secure Purchasing Transactions (CBID-Enabled)

In another example of secure shopping, the real-time knowledge of the identity of the wearer of the device allows financial particulars to be electronically exchanged each time an item is selected and purchased. This eliminates the need to repeatedly enter passwords, security questions, or account information for each transaction or group of transactions. As a result, this immediate purchasing system eliminates the processes associated with so-called online shopping "carts " because there is no need to cluster the items for the purpose of entering account information (see FIG. 8). For customer convenience only, groups of items purchased during the online shopping session may be treated as clusters or summarized to the buyer.

FIG. 8 illustrates a sequence of steps for performing a CBID-authorized online purchase (it is not necessary to collect items in an online "cart "). When an item to be purchased is identified 2900, an eye signal 2910 may be executed to indicate the desire to purchase. Once the item to be purchased is identified 2910, the CBID-based iris code 2930 is compared 2950 to the database of iris codes 2940 belonging to authenticated buyers. If a match is determined 2950 (and sufficient funds, permits, etc. are available), a purchase transaction is performed 2960. Thereafter, the device wearer may continue shopping for additional items 2900, and the purchasing process may be repeated a certain number of times.

According to another embodiment, systems and methods are provided for enhancing security and for enhancing streamline shopping in so-called "brick and mortar" retail outlets. In this case, a camera mounted on the headwear device may be used to identify objects that may be of interest to purchase, so as to view the environment of the wearer of the device. This identification may be based on a bar code or quick-response (QR) code typically attached to commercially available items. Such object identification uses image processing methods well known in the art.

Information about the article containing the proposed purchase price can be generated by the processing unit associated with the retail outlet. This information may be displayed on nearby monitors or on a head-mounted display associated with the device wearer. A CBID-based transaction may be initiated by the customer if the customer wishes to purchase the item's goods. These transactions can happen repeatedly throughout the store. A match between the delivered items and the transaction record may cause the items to be verifiably removed by the customer from the store. CBID-based retail purchases eliminate the need for cash registers or cashiers. In many cases, the display of automated, real-time information during the purchasing process also reduces the need for store clerks to help potential customers.

A particular exemplary implementation of the CBID approach is "buy at the aisle" using eye-signal methods or processes, which is referred to as "look-to-buy ". In this case, the purchasing process consists of looking up an identification symbol (e.g., bar code, QR code) associated with the item or an object identifier of the article to be purchased, and executing a purchase activation sequence. The identification symbols may be physically attached to the article or its packaging, or may be in a signage associated with the article (e.g., in connection with bulk articles in the barrel). The purchase activation sequence may include any combination of the following steps.

Decode the identification symbol (s) or object recognition with an index, which is used to retrieve information about the item from the database.

Display information (on the HMD or remote display device) containing the price for the item.

(E. G., Availability of similar items, quality assurance information) desired by the buyer using eye signals and / or other interacting means.

Optionally negotiate price.

● Use the eye signal sequence to indicate that a purchase has been requested.

● Ensure that the identified device user is authenticated and has sufficient funds to make purchases using CBID.

● Execute the purchasing process using secure telecommunications.

● Provide user feedback that the purchase is complete.

● Add the article to a "tally" or "virtual cart" of articles that can be exported from the store.

• Perform additional purchases using similar steps.

• physically "out" or "physically" export the item from the store so that the actual items or their equivalents can be exported or delivered in the near future.

● Permissions for goods to be exported from the store can be verified by visual inspection or automatically (eg, using the RFID method), without any cash register, Ensure that the items in the cart are matched.

This "look to buy" process has the following advantages.

● The ability to interact with store clerks is greatly reduced since the available detailed information is provided through the display, which can reduce or eliminate the need for store clerks.

• Purchasing information is stored and accessed in a central database, so that key elements such as pricing can be adjusted immediately and even negotiated with the buyer on an item-by-item basis.

• The retail infrastructure (ie, hardware) to implement the purchasing process may be minimized or non-existent (because the buyer supplies key hardware). Similarly, no retail hardware maintenance or upgrades are required.

• For "look-to-buy" purchases, there is no need for a cash register or checkout counter (including those who run them).

There is no ambiguity about items that may look or function similarly, since purchases are made when the goods are specifically viewed by the purchaser.

● Since all purchased items are individually identified, inventory control can be fully automated.

• "Look to Buy" can also co-exist with traditional retail purchasing methods.

A more generalized example of "Look To Buy" is "world is your store". In this case, object recognition is used to identify the items to purchase and they are simply shown in the user's surroundings. This can be based on eye signals and objects that are specifically observed using eye tracking in the user's environment. As an example, if a dress in a shop or a dress worn by another person is observed, one or more online sources for purchasing the dress can be identified using object recognition, and then the object, An instant purchase can be made mainly based on the perception of what is depicted on the display, or what is shown on the display (e.g., during a television commercial).

"World is your store" purchases can be processed in a similar way to online purchases. The main difference is that the specification of the object to be purchased is made on the basis of object recognition and can optionally be enhanced by staring and registering any identification tag (e.g., bar code, QR code) . In addition, these details may optionally be assisted by interaction with the device user. For example, if an important part of an object has not been observed, the user can see and interact with the objects to more completely and uniquely identify the identity of the object. As an example, purchasing a dress may need to look at the back of the dress where the front was originally seen. The user can be advised to look at the rear through an interactive process. An interactive dialogue that provides the user with text, graphical information, directional information, or "augmented reality" information padded within the user's field of view can facilitate the user's decision process, Interactively provide information through ocular signals associated with the capabilities, abilities, performance, quality, ratings, reviews by friends, prices, timing of shipment, and so on. In addition, such conversations may be subject to a substantially continuous or intermittent user authentication in order to verify user identity, block tracking of user actions on specific information, and so on. The purchase may also include access to a database of personal basic information related to identity, past behavior, performance, preferences, and desires, such as a device user's dress size, and so on. As during online purchases, the authentication for the purchase (s) is based on the CBID. The item (s) may be shipped to an address associated with the user.

Alternatively, delivery locations may also be specified based on automatic recognition of locations or individuals. For example, a delivery can be made to a personal address associated with the person being viewed by the device user. Delivery can also be specified based on the perception or details of the location associated with the wearer of the device (e.g., using GPS). Place recognition can be based, for example, on object recognition for street signs or buildings.

Other CBID-Enabled Secure Applications

According to yet another embodiment, apparatus, systems and methods are provided for controlling the exchange of secure data between a device wearer and a source of security information, while preventing sensitive information from going to unwanted recipients. In the highest level of expertise attempting to gain access to the transmitted data, the potential attack mechanism will attempt to mimic the bi-directional communication between the CBID system and the security server. For this reason, the communications of the remote computer with the CBID system are encrypted. The cryptographic key may comprise a respective headware system manufactured, a unique identification code assigned to the target iris code, and may be time / use-sensitive. As described above, the encryption and decryption processes within the CBID system may also include hardware components that are difficult to access and further increase the security of the transmitted data.

Examples of behaviors in which eye-signal control sequences utilize authentication include withdrawing cash from ATMs, making online purchases, verifying by identity, using eye-signal sequences, Compliance with identity verification requirements during any type of testing of documents requiring electronic signatures, online, personal, group, or other testing, for example, professional driving, piloting or other Compliance with the performance requirements associated with the identity for various forms of employment, such as transportation, etc., confirmation acknowledgment of the consent received orally or read by the user Thereby, a substantially continuous identification occurs during the saccadic activity during the reading. ), Acknowledging any legally binding agreements, observing identity verification requirements while collecting data for clinical trials or other medical research, treatment or testing programs, reading, checking-in , Any court requiring action - to comply with identity verification requirements during an ordered act or during a behavior by a parole, probation, or prisoner in prison or other form of receiving facility; Including, but not limited to, monitoring staff, agents, or Transportation Security Administration training or activities, combat or intelligence training activities, or census training or activities. Compliance with identity verification requirements for soldiers monitored or tested during the performance of the Identification during handwriting, including instances of using eye-signals associated with capture of the process, identification during competing or professional video game play involving eye-signal initiation and interaction, Identifying during a personal self-development program that includes identifying, weight loss, sports training, phobia overcome, speechlessness, smoking cessation, rehabilitation programs, tagging the identity during eye- (For personally identifiable follow-up access to the information), verifiably identifiable Participants in information sharing related to cases of mutual gaze in which the parties choose to share information ≪ / RTI >

According to yet another embodiment, devices, systems, and methods are provided for replacing or augmenting conventional password-based access to computing devices. Such a system can provide superior security compared to password and / or security questions. This is not only a result of the nature of the biometrics of the user identity (i. E., Completely unique to the individual), but also the ability to seamlessly monitor the device wearer substantially continuously within its self- contained system It is the result of the ability. The system may be adapted to perform saccades during different eye-activations, including observing biological or inanimate entities and behaviors, approximately every second or, if possible, in formations of the ocular signal or in a real or virtual environment. , And is designed to substantially re-verify the identity of the wearer of the device substantially at any desired frequency, including sampling rates that are sufficiently high to be re-verifiable during micro-saccades. Unlike traditional password input, which may be replaced by a user (either openly or unintentionally) after entering a password, the CBID-based computer access system can immediately stop security operations if a friendly device-wearer's identity is lost have.

According to yet another embodiment, devices, systems and methods are provided that enhance the security and convenience of so-called "digital" or "electronic" signatures. The digital signatures are used to indicate that 1) the source of the message or document is from a known sender, 2) the sender can not deny the message transmission at a later time, and / or 3) Lt; / RTI > The generation of a CBID-based digital signature provides a traceable way to verify a unique individual as a true source of the message. One example of a digital signature includes any form of productive creation for an individual, such as simple biometrics, typing, writing, speaking to form an artifact, As a result, the user's gaze can be focused on the production behavior, as well as the substantially continuous biometric authentication.

In addition, verifying substantially the identity of the wearer of the device allows a continuous series of traceable documents to be generated. If these electronically signed documents are accepted as legally binding, the ability to quickly exchange legal documents allows many professional service providers to ensure that more of their services are performed securely online . Examples of such service providers include financial advisers, insurance companies, lawyers, physicians related to telemedicine-based medical diagnostics and prescriptions, and real estate sales.

The communication link 155 may include wired Ethernet or other mobile communication protocols such as those described in IEEE 802.11, wireless technologies such as Bluetooth, Zigbee, and LTE, GSM, CDMA, and GPPRS have. Typically, multiple communication media and protocols may be involved in transmitting the wearer identity state and other data to a remote secure processing unit. For example, wireless communication (e.g., IEEE 802.11) may be used between the headwear device and the local router, where the local router transmits packets through a so-called twisted-pair wired system, - The pair wired system transmits data over a fiber optic cable to a remote central processing unit (e.g., a server system).

Some applications require verification of the identity of the wearer of the device, coupled with unidirectional sourcing of secure (i.e., encrypted) information to the external processing unit. Entering a credit card or account information to make an online purchase is an exemplary application of this mode. Other applications may require exclusively to receive security information only when there is a friendly identity verification of the wearer of the device. The display of classified, read-only documents is an exemplary application of this mode. Other applications may require secure transmission in both directions if there is a friendly identification of the wearer of the device. It is an exemplary application of this mode to protect both questioned questions and user responses to academic tests.

In yet another embodiment, systems and methods are provided for restricting recipients of security information to a limited number of identified individuals. For example, it may be desirable to distribute proprietary business documents to a group of identified individuals (without the possibility of the document being recorded or stored for some of the recipients). In this case, the documents are transmitted in an encrypted manner, along with enclosed iris codes of embedded device and device wearers and / or decryption keys linked to specific headware systems. It will be possible to decrypt documents only within a limited number of receiving users / systems if there is a friendly CBID and also optionally if the individual field of view breaker (see below) is in place.

In yet another embodiment, the CBID may be used to verifiably document the performance of behaviors by the identified individual (or group of individuals). In this case, when reading and / or viewing a particular set of data, the video sequences captured by the scene cameras oriented in the general orientation of the device wearer's field of view and / or the audio sequences recorded by the microphone are recorded. These data are accompanied by substantially continuous identification and time-stamps on the wearer of the device. Thereafter, the sequences may be obtained remotely (as needed) for a short period of time or for an extended period of time as documented evidence of the fact that the action (e.g., reading, viewing, etc.) Lt; / RTI > device.

Examples of applications for these embodiments include, but are not limited to, handling expensive materials (e.g., diamonds), improving assembly line checks (e.g., if a defective item has been removed), training processes (E.g., education), medical procedures (e.g., steps performed during surgery), catastrophic events (e.g., the identity of the driver involved in the collision) (E. G., Pre-flight testing by a pilot), and so forth. ≪ / RTI >

Other examples of applications include capturing the eyes of a professional with virtually continuous or intermittent authentication, such as celebrities, professional athletes, professional video gamers, surgical surgeons performing surgeries, or other professionals performing acts And simultaneously performs scene camera recording or virtual display recording. The resulting "gaze-cast" (i.e., recording of an external video in which the user's gaze in the form of being focused on the original or related objects or areas is simultaneously superimposed) , Or sold, licensed, or rented to individuals for other uses.

Degrees of Security

In other embodiments, the systems and methods may be adjusted to a wide range of security levels, including situations where extreme security is desired. Extreme security levels may be applied during military applications, for example, to obtain access to high value facilities (e.g. nuclear stockpiles) or covert operations. In these extreme security cases, the system must defend complex cheating attempts, such as placing an image of the iris or an extracted eye within the field of view of the iris tracking camera. As an example of the means against these attempts, the physiological responses of the eye can be monitored.

Security protocols may require multiple individuals to be associated with a given action or series of eye signals to unlock, activate, or authenticate an event. Individuals wearing HMDs with CBID and eye tracking must follow a security protocol that involves performing authenticated parallel, serial, or interrelated actions to initiate an event.

In embodiments having such a high security level, security-based measurements may be merged into a single measure of confidence that the surrounding environment is secure.

A composite security index (CSI) may be used to assess whether the security information is to be displayed and / or trusted. For example, a reduction in CSI may cause additional security questions or "dummy" information to further test device user responses. If the CSI continuously drops, various additional steps may be taken where, for example, the CBID platform may be requested by a remote security server to transmit video and / or audio about the wearer ' have. Depending on the desired security level, the components contributing to CSI may be selected narrowly or broadly. The CSI components may include one or more of the following.

a. The degree of matching (including strictness criteria) of identification coefficients associated with real-time iris scans,

b. Highly strict (i.e., on-axis) elapsed time after iris recognition,

c. The degree of reliability of pattern matching with areas around the eye during an off-axis gaze,

d. Ocular biometrics including geometry, position, and motion of iris,

e. Looking at the direction associated with the display, the time (and consistency) spent by the device wearer,

f. The presence, frequency, and velocity of eye flicker (i.e., the short duration that the eyelids cover the iris)

g. Outputs of sensors that determine the structural integrity of the headwear device

h. An elapsed time measurement associated with device wearer feedback responses, which is compared to the display time of the information (e.g., questions) on the head-up display or nearby display monitor,

i. Changes in ambient light conditions, including pupil responses within the normal range for these changes,

j. The timing of pupillary light reflections (by adjusting the illumination) caused by the headware system at random intervals,

k. Whether the images viewed by the scene camera are generally consistent with the predicted environment (e.g., an obvious attempt to change the head monitor versus viewing the display monitor)

l. Whether the sounds monitored by the microphone are generally consistent with the predicted environment

m. The degree of matching of coded video patterns originating from a security information source and displayed on a nearby monitor, as compared to images subsequently viewed by the scene camera

n. Input of identification codes known only to the wearer of the device (and which may be time-sensitive).

An example of using biometric items f and j described above is that a camera that is aimed at the eyes may not be able to detect an imposter photograph, a contact lens, or even an eye that has been surgically removed from the owner Instead of a replica of the eye in the form of an eerie, it can be to ensure that you are really seeing the real eyes of the user you want to be authenticated.

In a further embodiment, ocular signal discrimination and authentication is custom-built silicon (i. E., Designed to interfere with attackers by constructing identification and authentication methods in silicon so that logic and functionality remain within the lowest levels of the system architecture) Custom semiconductor ASIC). This helps defend against security attacks based on changing the device's programmable software. Moreover, methods are available for encrypting firmware or hardware encoded instructions within such low-level system architecture.

"Eyes Only" Viewing by an Identified Individual ("

According to yet another embodiment, systems and methods are provided for confining the receipt of security information to only one substantially persistent identified individual, without the possibility of anyone recording or storing secure content for presentation to others . For example, during the remote management of on-line tests, the test recipient needs to read the test questions. However, if there was a way for another person or recording device (e.g., camera 125 in FIG. 1) to view test questions to store or transcribe test questions with the intention of revealing the test questions to others, The integrity of the test management for the (or similar) test may be undermined in the future for others to take the test.

For these applications, the CBID system can be implemented with a head-mounted display. One or more field-of-view breakers are placed around the display, in order to limit the ability of the display to be visible only to the wearer of the device. The key aspect of this configuration is the fact that both substantially continuous user identification and display of security information are performed within the same device. Security information is displayed only when a friendly CBID is established, which makes it impossible for other individuals to view security content. Any object placed on the light path between the head mounted display and the eye of the wearer of the device destroys the CBID.

In additional embodiments, additional security levels can be established by ensuring that the field-of-view breakers are in place and by adding sensors that ensure that the components of the CBID system are not tempered. A relatively simple way to implement this is to add in series conductive paths through all the structural and electronic components of the system. If manipulated or any system element is removed, the electrical continuity is broken and thus can be detected by the system. As a result of this information leak, the friendly CBID will be disabled and / or an indication of an information leak will be sent to the security information server.

The structural integrity of the field-of-view interrupter surrounding the head-mounted display can also be used for a wide range of applications including, for example, contact switches, push button switches, hall effect switches, reflective photocells, and / Lt; / RTI > range of electronic components. The field-of-view breaker may be made for dual-use, for example, it may be unobtrusive during non-security applications, so that the head-mounted display is in a see through mode , A state in which the breaker is removed) or a high security "individual viewer" mode (i.e., with a visual breaker attached).

In another embodiment where extreme security is required, the head mounted display may be fabricated with a highly manipulated-view field-of-view breaker that is permanently disposed in place. In another embodiment where extreme security is required, the head mounted display may include a layer of added or integrated electrochromic material, wherein the transparency of the electrochromic material layer may be electronically controlled .

7, the remote processing unit 160 and the device 400 used to display the information are located at a predetermined distance from each other and that require a secure (e.g., encrypted) communication link 155 They can be separate devices. Alternatively, the remote processing unit 160 and the display device 400 may be integrated into a single package. One example of such devices with integrated processing and display capabilities are laptop computers, tablets, and mobile phones where software can be coded to decrypt secure data only when there is a friendly user identification.

9A and 9B illustrate an exemplary system integrated into an eyewear device 500 that includes a head-mounted display 505 (head-up display, HUD, Display, virtual reality glasses, augmented reality glasses, or smart glasses). As shown in FIG. 9A, the system includes a visible or near-infrared light source 400, which illuminates the area of the iris 115. A camera 125 is used to acquire images of the iris 115 and features in the area of the eye. The display is configured such that the structured light (i.e., constituting the display image) is reflected or projected onto the retina 52. The eyewear device includes a nose piece 510 and an ear stem 515, which allows the system to be worn in a manner similar to conventional glasses. The CBID system 100 may be integrated into an eyewear device or may be fabricated as an accessory to a wearable computing or eyewear device.

In the case of FIG. 9A, the display 505 is configured so that the wearer of the device can look over the projected image to see the surrounding environments. In FIG. 9B, the viewfinder circuit breaker 520 is strategically placed so that no individual (or video system) except the wearer of the device can see the content of the wearable display 505. As a result, the wearer of the device can not see the surrounding environment with the same eyes used to view the display 505. This configuration prevents the content of the display 505 from being moved or recorded. Switch 525 and / or other sensing (not shown) to detect whether there is an attempt to disable the concealment function of the visual field breaker 520 and / or whether the visual field breaker 520 is in place and / Elements can be used.

As an alternative to the "mechanical" field-of-view breaker as described above, other exemplary embodiments may include a head-up display (or a near-eye display) in the area between where the content is displayed Lt; / RTI > and other display devices). It acts like an "electronic shutter" and can control whether any person or recording device in the external environment can see the content of the display or not. Electronic controls for optical transparency or electronic discoloration can be implemented by a number of mechanisms, including liquid crystals (i.e., the same principles used in LCD devices), suspended particle devices (SPDs) , Nanocrystals, and other mechanisms often used in so-called "smart glasses ".

Controlling the opacity of the material (s) around the HUD, similar to the detachable mechanical circuit breaker just described, prevents the display's content from being seen by anyone other than the wearer of the device (see Figure 9b). The main advantage of this electronic control is the capability of the device itself to control opacity, for example, it can block light transmission only when the device knows that security information is being displayed. If not in secure mode, the HUD can operate in normal display mode, so that in normal display mode, the user can see the actual environment "beyond" the display. In the secure mode (i.e., opaque), the user can not see the outside world beyond the display, and more importantly, no one in the outside environment can see the content of the display. When the components performing the display function and the components performing the blocking function are combined to form a structure (the structure can not be separated without destroying the function), this strategy is particularly useful .

As an additional mode of operation, the light-blocking or opaque mode can also be used to minimize interference light from the outside world, which is intended to improve the viewing of fine detail in the HUD. For example, when pictures with fine, black, or low contrast structures are displayed, when interfering light (i.e., so-called "noise") from the surrounding environment is blocked, these structures can be better seen by the device wearer have. Electronic control of opacity can allow this type of control to be performed automatically (i.e., depending on the display content) or under user control.

As a further refinement to this embodiment, it is also possible to control the opacity of certain areas of the HUD (not the entire display area). In such spatially accessible opacity control, the device may block only one or more areas around specific security information. For example, an area around the account number or user password may be blocked from the outside line, while the user may continue to view the outside world in areas other than the security areas. Accessible spatial control over opacity is particularly well suited (and well developed in the electronics industry) for LCD technologies (backlit-free) where similar mechanisms are being used in LCD projection devices,

Secure Personal Augmented Memory (PAM)

Is a system that utilizes eye tracking as a component for electronically enhancing the medium with a long-term memory of the wearer of the identified device. PAM storage and retrieval may include completely interchangeable information modalities (images, audio, video, text, icons, etc.). The following are three overall steps for PAM.

1. Identify trackable objects that are important to the individual.

2. Keeping a dynamic, historical archive of multiple-aspects, in conjunction with data sets containing one or more identified trackable objects or attributes.

3. Identify and retrieve one or more data sets based on any mode of traceable object specification (s) or attribute (s).

The first step in PAM is to create records of traceable objects (which are "important" to the identified individual). These trackable objects can be in various modes including images, audio clips, video clips, text, icons, tags (e.g., price, manufacturer), objects on a location-based tagged map, Transformation algorithms can be used to combine all of the different storage modes associated with each trackable object. For example, image recognition can be used to identify (e.g., convert to text) objects within one or more images, speech recognition can be used to convert audio to text, and text-to- The text may be used to convert the speech to speech, the icon may be associated with a particular entity or group of entities, and so on. In addition, the regions of interest represented by the line of sight can be associated with iris code-based authentication associated with, or coexisting with, the region of interest, to reduce scene processing requirements and identify information with only relevant user-identified objects.

If necessary, further transformations of information can be applied. For example, text or audio can be converted from one language to another. Image recognition can be extended to identifying specific objects. For example, the algorithms for identifying the vehicle may be configured to further identify the specific vehicle (s) of the wearer of the device. The price and physical characteristics (materials, design features, etc.) of an object can be identified and included in archival algorithms.

Individual users can create any number of personalized, traceable objects. Most commonly, gaze tracking is used to identify objects that are being viewed in a single environment. Traceable image objects may also be identified from downloaded images or photographs (e.g., road signs, Eiffel Tower, etc.), where gaze tracking is once again used to identify the objects being viewed. The objects shown can be identified and indexed, for example, based on image recognition. Other aspects (e.g., voice phrases) may be similarly categorized (e.g., recognized) and indexed for subsequent searches.

The second stage of PAM dynamically adds an indexed, searchable historical archive of the objects shown, according to their associated aspects, which may include images, video, text, audio, ≪ / RTI > Additional information available at the time of creation of the record may also be stored to define additional object "attributes ". This may include, for example, the date and time of creation, the user and / or device that created the data set, the geographic location, the presence of other identified individuals, and so on.

As a final step that may be repeated any number of times, a search for indexed objects and their associated data may be based, for example, on viewing the object or similar object. Criteria for information retrieval may include, for example, voice recognition, a portion of text, an icon associated with a particular object, or a classification of objects, etc. In further embodiments, the information mode initially used to store information need not match the mode used to retrieve the information from the data archive.

For example, an image of a wallet can be used to identify a particular wallet at a GPS-based location. Data retrieval for wallet, "Where's my wallet?" In response to a verbal query such as < / RTI > Subsequent information retrieved for the purse may take the form of a directional map (i.e., other data type) that shows the location of the purse in relation to the current location of the device user.

The search for a particular, desired data set can be further separated based on the attributes of the objects. As described above, the attributes include supplemental information stored for the object. If the attributes include location, date / time, and price, the PAM system can answer the following questions: "What is the cheapest shirt I saw at the department store last Tuesday?"

If two or more data sets match the search criteria, then the device user can either: 1) separate further based on additional input using any mode (e.g., see closer to the object, speak the words, , Etc.) or 2) thumbnail sketches of possible desired data sets (e.g., in reverse chronological order).

Another exemplary scenario using PAM is the ability to continue tracking newly viewed objects (e.g., a set of desk drawer keys). Device users can start by looking at the keys and saying "My desk drawer keys". Each time the keys are viewed, a data set is stored that contains the location and time at which the keys were viewed. Later, "Where is my desk key?" The device may respond with an image containing the location where the keys are located and the last time the keys were viewed.

As another example of PAM, where the initial event triggering PAM data storage is a "virtual object ", a user can view an individual as an online image or a person's video form. The person can be identified through online information or by mentioning the name (e.g., "Jim Smith") when viewed. Next, the user may mention any other desired attributes (age, occupation, etc.). Subsequent data sets may be attached whenever the individual is viewed directly or whenever viewed in a virtual manner. Later, the user may ask: "When was the last time I saw Jim Smith?" In addition, the device can identify Jim Smith whenever he is viewed on a scene camera.

Another example of using PAM is initiated by looking at an individual's car and saying "my car". When the car is parked, the car is identified and a data set is generated that includes the latest images and location of the car. Later, the device can respond to an icon on the screen, which answers the question: "Where is my car?"

If "too much information" becomes available, the device user may incorporate certain data sets and / or forgetting mechanisms on the target objects. For example, the data generation time can be used to "forget " old items, such as a particular key package or a car's past locations. Different criteria can be assigned to different traceable objects.

Both information storage and information retrieval can be based on CBID. The data sets may reside on a wearable device, in a remote processor, and / or distributed over cloud-based systems. Regardless of the archival location, both the storage and retrieval of such data may be confined to the CBID source of the PAM, or may be limited to a group of individuals authenticated by the CBID source of the PAM. With CBID, separating information into individuals can be limited to that individual, even if one device is worn by many individuals.

As described in Systems and Methods for Biomechanically-Based Eye Signals for Real and Virtual Objects for interacting with real and virtual objects,

Eye signals (in particular, eye signals associated with machine learning techniques) can be used to determine a user's intent.

Using these methods, PAM-based information can be retrieved and can also be "suggested" to the device wearer. This is a more abstract data retrieval method (as opposed to solving certain questions). However, the device can present the archived data based on objects seen in the environment, given questions, time of day, geographic location, recognized words, and so on. Future "proposals" may also be based on changes to machine learning approaches based on whether the device wearer has accepted or rejected "suggestions ".

Secure Human-to-Human (s) Communications (EyeConnect)

When we meet people, whether as a species, business or social, we have a common connection or share a domain of interest, and we are engaged in communication. The process of determining whether to communicate, when, how, and what to communicate with is inherently inefficient and often socially awkward. Also, human memory is imperfect and we often encounter individuals who have forgotten details of past encounters (eg, names or topics of last encounter). All of us are sensitive to the social awkwardness that arises from these encounters, and that degree hinders social discourse.

Processes and applications, referred to as "EyeConnect, " are a method for wearable computing users who, while maintaining the explicitly selected and well-controlled security and privacy levels, It is a way to make it efficient. EyeConnect provides improved connectivity and communication among individuals within each other's eyesight, with simple, mutual eye contact. CBID-based iConnect provides privacy for individuals by establishing connections between CBID registered individuals, for example, where security and privacy levels are established.

Examples of embodiments for such applications include the following.

1. For those who know each other but have forgotten the details of their last meeting, EyeConnect will provide the forgotten name immediately upon the first eye change and will be able to provide context for the in-person encounter To provide relevant information smoothly.

2. For those who first met at business, educational, political, social events or at other places; Meaningful, relevant, and fruitful connections are often left to luck and take time to develop. Although systems and procedures for automating or enhancing connections between individuals exist today, they are ineffective, inaccurate, ambiguous, and sometimes awkward, or in an existing communication flow and social exchange. It can also hinder. IConnect enables secure levels of efficiency and elegance in establishing new connections in these meetings.

3. For those who first met through the introduction of an individual (ie, a third party registered with the CBID), iConnect facilitates the exchange of relevant information in a contextual context, Issues, and so on.

4. For a group of individuals gathered for business or social exchange, iConnect facilitates the inclusion and inclusion of individuals in dynamically formed groups for the purpose of information sharing.

The range of information aspects that can be exchanged along the iConnect also extends the value of iConnect. Not only visual information is shared immediately and separately on the display (e.g., the HUD). Aural information can also be exchanged as an instant "phone call" that can be established between two individuals who have exchanged their gaze in the distance. This may be useful in certain security applications, social situations (eg, party, bar-scene), and may be useful in other groups that want to spot one another and communicate remotely It can be useful.

In the exemplary embodiments, EyeConnect performs these functions while respecting any prohibitions on facial recognition or other object recognition software. However, the architecture associated with EyeConnect takes into account this eventual allowance of automated recognition of people in view of wearable cameras. In fact, the CBID ensures that the identity of the person obtaining the camera images is associated with these images and / or by ensuring that the identity is used to notify the person being photographed that these actions are taking place, Can help alleviate them. The CBID may allow or prohibit the storage and / or transmission of camera images only if the individual allows such images, based on the identity of the image acquirer. In addition, the relevance of the acquired arbitrary images to the identified device user must also serve to dissuade acquisition of inappropriate images. Repeated offenders must be identifiable based on their identities stored in image and video files.

In addition, the iConnect application supports geo-location and provides unidirectional or bi-directional information exchange of proximity-relevant individuals. In another exemplary embodiment, the two iConnect users stare at each other for a while to determine if they should be connected or already connected. These mutual gazes in each other's eyes for activation occur at the same time, and geo-location generates a "spark. &Quot; If the two users are not already child connected and have sufficient commonality, then the spark will ignite and users will be able to use contextual relationships to support personal, introductory conversation. And receives an immediate "Connect Alert" with the appropriate information considered.

By one person's eyes, by instantly interacting with the display after a connection alert, a connected pair of users can bridify selection information for each other. This can occur before, during, or after participating in a physical conversation with each other. In addition, the iConnect facilitates the sharing and transmission of additional information beyond what is initially shared through a connection alert, which one user has decided to provide.

There are two key elements that support iConnect. 1) the user's "Persona" and 2) the user's "Event Profiles". The user's persona may be entered manually, filled in automatically with information from linked accounts, such as Facebook or Linkedln, and / or the user's online footprint, Or other information from the < / RTI > Persona data includes information such as name, goals, interests, jobs, employers, family members, colleagues, phone numbers, hobbies, habits, favorite sports teams, favorite music,

The user's event profiles include information that facilitates both matching and sharing. The user establishes event profiles, starting from a set of standardized event templates, with fields filled with data from the user's persona. Event profiles can be populated automatically or manually. Following an "spark" (i.e., a coincident, local, interchanged line of sight), the iConnect will send two active users ' events Compare the data from the profiles. Commonality within the selected topics, thresholds based on connection interests, and / or other factors are used to determine when the ignition should occur. In addition to providing information for evaluating a match, each event profile also accurately specifies what information should be shared in the connection alert. Thus, event profiles are used not only to determine whether or not an ignition should occur, but also to determine which introductory information should be automatically shared.

A set of standardized event profile templates can be used for business meetings, parties, bar-scenes, classes, office places, vacations, street-scenes, sporting events, beach- scenes, churches, sales meetings, and so on. Each template may be customized by a user or group of users, and new event templates may be created and shared within the iConnect user community.

In addition, a device user can maintain any number of customized profiles to meet a wide range of day-to-day conditions. For example, a user may maintain profiles to cover the following situations: general business, a number of specific business meetings and other events, educational events, hobbies, social interactions, churches, etc. . By maintaining different profiles for different situations, the dissemination of information can be controlled very finely and can be controlled by family, friends, close friends, business associates, close business associates, teachers, And the like, in a manner suitable for the group.

Moving from one situation to another can be either 1) explicit (the user notifies the device that the situation has changed) or 2) implicit (the device recognizes different situations). Recognition of situational conditions can be based on consideration of several factors such as facial recognition, (familiar) object recognition (e.g., objects in the house), features (E.g., sports events, golfing), geo-locations (e.g., airports, hospitals), signage, events scheduled on the user's calendar, (E. G., Meeting attendance, hotel stay), presence of music (e. G., Concert), and so on. In some situations,

Two or more profiles may be valid at the same time (e.g., hobbies and social actions are taken at the same time). In this case, the exchange may be based on permissions in any applicable profile. In addition, the exchange may be "conditional" based on other factors such as the degree of overlap of interests and / or certain conditions (e.g., within a particular time zone, living in the same city) .

At all levels of interaction, the exchange of information is based on the participants' established identity and their pre-established privacy level for the exchange of information. For example, using the CBID, it would be impossible for an intruder to swap or pick up a device that was temporarily set down and to transfer personal, device-owner's childconnect information to a third party.

In another embodiment, two users who have previously met through the iConnect or two users who were connected to each other as acquaintances in the network of iConnect have been reassembled twice in an embarrassment of not remembering their names You will not face it. IConnect provides "invisible name-tags" and more. Two older acquaintances re-acquaint immediately with the help of both name memory and complete post-war contexts of previous encounters. This includes past meeting events, places, times, shared information, and even reminders of supplemental information logged in at or after the previous encounter.

The iConnect also provides instant online search results for connected individuals, with immediate presentation of contextually relevant information. Both of these users at the meeting are on level ground and are relieved by the innocuous rich memory improvements offered by iConnect.

In yet another embodiment, overlapping interests between two or more eye linked individuals can be identified and displayed. These concerns can be placed in the historical context of past meetings and information exchanges on identified topics.

As a scenario for helping the illustrated embodiments, device user 1 (U1) meets user 2 (U2) at the conference. They have never met, but both have worked in the same company in the past and know many people in common. They loaded their business-meeting event profiles for iConnect before attending the meeting. These event profiles not only contain their personal goals for attending the meeting, their career histories, the names of colleagues who were familiar with their previous work, as well as other information that they consider relevant for sparking child connections . They also organized their business-meeting event profiles to include their preferred hobbies, their wives and children's names, their children's school, and other personal data that could produce connections at a more personal level. U1 has set up his business-meeting event template to allow information sharing only on his professional background. The setting of U2 allows sharing of all matching items.

With the EyeConnect active, they stare at each other for a few seconds, standing on the waiting line and seeing if they will be connected. Each of their wearable devices displays a connection alert. Both read that the company has past overlap and U1 finds out that their child and U2's children are in the same school. They also know what each person attended in the meeting and eye-scroll other information that is optionally shared from their profile. Later, they engage in conversations, instantly share common areas of interest, and work hard to explain their purpose for the meeting. They exchange credentials through the iConnect application and make appointments to meet. In a matter of minutes, they finish their initial connection and move to a child connect with other attendees.

Extending the scenario to illustrate other embodiments, during the encounter with U2, U1 is also briefly associated with user 3 (U3). Later, U1 finds U3 at the party and thinks he / she is familiar. U3 and U1 stare at each other, generate a spark, and EyeConnect tells U3 and U1 that they are already in the eye connection. The Acquaintance Profile takes precedence over the Party Event Profile, and EyeKnowU alerts are generated for U3 and U1, respectively. U1 can know the name of U3, the annotations to other events they met, and the information that U3 shared in the event. U1 notices that there is an ignition with a new match for the current party event and chooses to view the connection alert for the party via eye-signaling. U1 looks at some additional information provided by U3 for the party context, and U3 is also a pilot. Note that U3 is the same. They meet with each other, acknowledge their past encounters, and begin to discuss their passion for flight.

The EyeConnect service can selectively apply face recognition to images of other individuals. Steps can be taken to prevent providing surreptitiously information about people within sight, especially in situations where there is a prohibition on these functions. When two people gaze at each other at the same time, EyeConnect's eye tracking software works in conjunction with a wearable scene camera to define spark events derived from information the user has allowed to use. For example, in the above example, eye tracking software in U1's EyeConnect application notes that U1 is staring at another person's eyes. U2's EyeConnect software also notes that U2 is staring at someone's eyes. Each takes a temporary, disposable picture of the face of the person being examined, so U2 captures the face of U1, and U1 captures the face of U2. Facial features may be extracted from each face for matching and the image is retained for user verification.

Each image is stamped with time and geographical location, and sent to the server for spark-testing. The image of U2 is matched with the time, face and location of U1 (using a pre-stored image of the U1 face provided to the eyeconnect by U1 when registering with the eyeconnect), and the image of U1 matches the time of U2, The server declares a "spark" between U1 and U2. As part of the connection alert, they are each provided with an original image for verification. Optionally, the persona image of each user may also be shared with a soliciting user.

The server then honors the privacy of the individuals, discards the captured temporary images and extracted facial features, and screens their "threshold of commonality " using their event profile information. If it is assumed that the connection threshold has been exceeded, the server declares a "connection" between them and an interconnect alarm is generated and the information as specified in their event profiles is shared.

Exceptional handling and additional features

Power consumption and available bandwidth are very common issues for all wearable devices. If EyeConnect is possible, gaze-tracking can be performed within a defined field-of-view (FOV), for example, for one second or other desirable periods. For each captured line of sight, a gaze point is mapped onto a limited area-of-interest (AOI) in the FOV of the wearable camera. The iConnect application looks for faces in these AOIs. If a face is found and the face appears to wear a headwear device, the image of the face is transmitted to the eyeconnect cloud to determine whether the user participates in a mutual gaze . There are many optimization opportunities for acceptable device latency while maintaining reduced power consumption because of eye-initiated Connect requests, the proximity of other active eyeconnect users, Or pushed or pulled geo-location data sent to the headwear device, methods for optimizing image processing for faces wearing headwear devices.

As yet another exemplary embodiment, the iConnect is capable of providing low-accuracy tolerance along with graceful degradation of performance resulting from reduced accuracy or loss of identity and / or tracking within a large FOV on a wearable computer. It may be designed to operate with tolerance, or even temporary eye tracking may be inoperable. Eye tracking technology helps to select one face from among images. In addition, the selection for a pair of users is screened by time and geo location-stamping.

As described above, the eyeconnect extracts the area of interest (AOI) based on the user's line of sight. The size of the AOI is determined by the estimated eye line accuracy. For example, a ± 10 degree gaze point accuracy of ± 10 degrees may require a 20+ degree radius AOI for a 20+ degree radius for image analysis. If the system experiences total loss of eye tracking, AOI will be the overall FOV of the scene camera. This would require more processing power to identify a mutually-gazing device-wearing face

Screening and recognition of mutual-gazers may also be enhanced using computer vision (CV) methods that screen device-wear individuals on an ongoing basis. The devices may be controlled via controlled visual information, such as, for example, one or more external directed infrared (IR) LEDs and / or certain recognizable patterns (e.g., visible light or infrared) attached to the device and / These CV methods can help. IR LEDs can be modularized (by encoding specific modulation patterns) to avoid IR-based eye tracking or collisions with other devices or to identify a particular individual. Other types of modulated / distinguishing information exchange may also be included.

As another exemplary embodiment, consider the problems of potential concurrent misconnections between the iConnect users, which are rarely co-located with multiple concurrent devices Let's consider four users who are wearing an iConnect and are active. Ul and U2 sit side by side and U3 and U4 sit next to each other (Ul and U2 are sitting side-by-side, and U3 and U4 are sitting next to one another). U1 and U3 join the gaze in hopes of a connection, and U2 and U4 also look at each other. If the accuracy is incorrect, the system may think that U1 and U4 stare at each other and think that U2 and U3 stare at each other. In this case, the system can provide various mechanisms for solving such a problem.

1. EyeConnect can notice the possibility of errors and alert users.

2. The iConnect can make optimal guesses and request confirmation with easy calibration available.

3. The iConnect can ask each potential user to choose between potential target individuals.

It should be recalled that when two users look at each other, a snapshot of each user's face is temporarily maintained, which provides information for verifying that two users are actually looking at each other will be. As previously indicated, the availability of these snapshots and the provision at the appropriate time can provide confidence for the intended connections. Existing images from the snapshot or the persona of the target may be used for this purpose.

These exemplary variations, where accuracy leads to further eye contact from one additional device-worn eye-connect user, can also be addressed with a combination of more processing of large AOI combined with appropriate information in the user interface. Regardless of false-positives or false-negatives, the potential rate of error increases with decreasing eye-tracking accuracy and with more EyeConnect users.

The following are additional exemplary features, and one or more of them may be included in the iConnect systems:

1. EyeTalk - EyeConnect supports "voice connection".

2. Geographic location metadata - Quick information that is visually provided, this information includes: a) proximity contacts (known to nearby people), b) potential iConnect users who are close to the connection (where to find them, Why it should be connected, etc.), and c) easy reminders that include a list of name reminders and pictures.

3. EyeNotes - notes to acquaintances, which can be voice-transcribed, added as text, or based on screened access to Facebook, LinkedIn, and so on.

4. Pre-Event Connections - Connection alerts can also be allowed as soon as the user loads the event profiles and specifies the time and location for the event.

5. Eye-No Signaling Actions - General information about people based on any form of identification, including geographic location.

6. UniSpark - Eye-signal selection of other device wearers who opted-in to allow matching and / or access to their persona or event information. For example, user 1 (U1) may take user 2 (U2), another device wearer who has previously agreed to allow matching (one-way or two-way sharing) with a soliciting user via an iConnect . When a match occurs, U1 receives a connection alert (ConnectionAlert) along with information about U2. U2 can also optionally receive a connection alert. If the eyeconnect is active for U2 and U2 has enabled non-gaze alerts, U2 enables a Gazeless Connection Alert. If U2 is not driving the iConnect, U2 may receive a subsequent report on any of the established connections with the request, the logged connection alert information from the matching users.

7. AnySpot - In places, times, and / or situations where facial recognition is allowed, the device user will be able to acquire information regardless of whether the user is using wearable computing / For that, you can search for any face. Using facial recognition, device users receive any available information from any source, including their own contacts for each individual they see. IConnect provides display-based information, including AR presentations of information, and supports unidirectional information access (including recalls, notes, etc.) and two-way information exchange.

The following is a list of additional exemplary connection settings, along with the characteristic elements in each setting. All implementations may be CBID-based and may accommodate displays for eye tracking, and non-augmented reality / virtual reality, augmented reality, and virtual reality situations as well as contact lenses, pawns, and / can do.

Optionally, the Gaze-Assisted Object Identification Optionally Assisted by Dialogue

Computer vision techniques for identifying objects from a still image or a series of rapidly changing images can be used for a wide range of applications including significant processing capabilities, high bandwidth image data transmission for non-local processing, ), Non-real-time processing, and wearable, camera-equipped computers that are limited by limited processing capabilities, camera resolution, transmission speed, battery power and / It is common to require some combination of different components. Recent developments in the field of eye tracking technology have enabled the development of low-power, low-processing, self-contained wearable technologies. The technique is efficient, focused, and distinct (i.e., private). Within the scene, adding a human cognitive selection to specify a region of interest for a rapidly changing series of images dramatically reduces the resources needed to visually identify the object in different contexts . It is also desirable to have a visual feedback system that enables intelligent interchange during a computer vision search, which combines human cognitive processing and machine processing fluidly, intuitively, and rapidly so that the wearable computer can access the context To identify the objects.

A "dialogue " between the software and the user can be used to confirm object identification or can contribute to object identification. Once confirmed, objects can be used for eye connections and information sharing.

Structured Exchange of Personal Information

Persona-based matching to allow and facilitate selective information sharing may include: real-time use of wearable computer systems with or without displays, proximity (geographic location), and event-based profiles for specific sharing . Other indications to share include cross gazing, one way gaze, handshake, and / or other signals (e.g., body language) or gestures. The systems may include local or remote information management, screening, and / or transmission of user-specific information. In addition, the systems may include community sharing, refinement, and evaluation of event profile attributes that may include specific screening criteria and sharing criteria.

Event-Based Selection of Information to Determine Relevance (Determine Relevance)

Event filters can be used to determine the relevance of personal / interaction relationships. These include user-loadable, calendar-loadable, filters generated by algorithms, mutually or collaboratively selected, and / or dynamically loaded criteria can do. Dynamic filters can be loaded before, during, and after an event.

In this paper, we propose a methodology for identifying individual participating parties through simultaneous gazing (Engaged Parties through Simultaneous Gaze)

The system can use wearable devices to compare images of individuals gazing at each other. The system may utilize event profiles, timestamps, and / or geographic location; And facial recognition software can be avoided (except for temporary or priori images). The system may be an alternative screening method, which may include an identification of users who "wear the devices. &Quot; Methods for this may include: computer vision to identify wearable devices, users seeking / participating in a cross-reference, IR, and / or other signaling protocols between devices when another device looks eye active.

Mutual Gaze-Based Protocol for Event Initiation

A number of different criteria and heuristics can be used. And may be used to initiate and expand (or vice versa, maintain privacy) admission-based events. Of these, it is important to detect simultaneous eye contacts for initiating an event. Other standards include wearable systems, camera-based conferencing systems, 1: 1 staging, 1: n staging, n: m staging (grouping of events for information sharing), 1: 1 private sharing within large groups, 1: n private sharing within a larger group, and / or the presence of a critical time to define an eye-contact. In addition, the criteria may depend on whether other actions are involved, such as additional screening to obtain event qualification, whether the event can cause some type of control (e.g., event registration, change of information ), Whether the event can cause one-way transfer of information from one party to another, or two-way transfer between the parties, and / or whether the action is real-time or delayed.

Common Gaze Contribution of Data for Simultaneous Location and Mapping (Simultaneous Location and Mapping)

The user's gazing can be coordinated and directed among the users through guidance and can be fed back from the wearable displays. The displays may be in-line or peripheral to the device. For example, the gaze of multiple users may be directed to a playground screen. Confirmation and / or analysis focused in a particular direction may be determined from users at multiple geographic locations.

Common Gaze Initiation of Multi-Mode Information Exchange < RTI ID = 0.0 >

Eye connections may include other aspects of information exchange, such as visual information exchange (e.g., video or images) or real-time voice information exchange.

For convenience, operations are described with various interconnected functional blocks or separate software modules. However, this is not necessarily the case, and there may be instances where these functional blocks or modules are equally combined into one logical device, program, or operation with obscure boundaries. In any case, the functional blocks and software modules or the described features may be implemented on their own or in combination with other operations, either hardware or software.

The foregoing description of the exemplary embodiments has been presented for the purpose of illustration and description. It is not intended to limit the invention to the exact form just described. Many modifications and variations of the embodiments described herein will be apparent to those skilled in the art having referred to the foregoing description.

Further, in describing exemplary embodiments, the present disclosure may represent a method and / or process as a sequence of specific steps. However, the method or process should not be limited to the specific sequence of steps described, so long as the method or process does not depend on the specific sequence of steps described herein. Other sequences of steps are also possible, as will be understood by those skilled in the art. Accordingly, the specific order of steps described herein should not be construed as limitations on the claims.

It should be noted that the elements or components disclosed in any embodiment are illustrative only of specific embodiments and may be used in combination with other embodiments.

It is to be understood that the principles of the invention have been described and illustrated in the exemplary embodiments, and that the invention may be modified in construction and detail without departing from such principles. The claims are intended to cover all such modifications and variations.

Claims (50)

A system for providing a substantially persistent biometric authentication for a user,
A headgear configured to be worn on a user's head;
A plurality of cameras mounted on the headgear, the plurality of cameras being directed from the different angles toward the eyes of the user when the headgear is worn to monitor movement of the eyes;
A processing unit operatively connected to a detector and an electronic display,
Analyze one or more images from the cameras to identify the iris of the eye;
Select a first camera from the plurality of cameras based on the one or more images, the first camera providing a desired view of the iris;
Analyze an image from the first camera to identify features of the iris;
Identify the user based at least in part on features of the iris; And
And when the identity of the user is confirmed, permitting the user to perform one or more actions. A method of providing a substantially continuous biometric authentication for a user wearing a headgear, the headgear comprising a plurality of cameras directed from the different angles towards the user's eye,
Analyzing one or more images from the cameras to identify the iris of the eye;
Selecting a first camera from the plurality of cameras based on the one or more images, the first camera providing a desired view of the iris;
Analyzing an image from the first camera to identify features of the iris;
Identifying the user based at least in part on features of the iris; And
Allowing the user to perform one or more actions when the identity of the user is confirmed
The biometric authentication method comprising the steps of: 3. The method of claim 2,
Wherein the first camera is selected based at least in part on the shape of the iris identified in the images from the first camera. The method of claim 3,
The first camera is selected based at least in part on the fact that the shape of the iris identified in the images from the first camera is substantially closer to a circle than the shape of the iris identified in images from other cameras The biometric authentication method comprising: 3. The method of claim 2,
Wherein the first camera is selected based at least in part on lighting contrast in the images from the first camera. The method according to claim 1,
After allowing the user to perform one or more actions,
Analyze another image from the first camera to identify features of the iris; And
Identifying the user's identity based at least in part on features of the iris from the other image
The biometric authentication method further comprising: The method according to claim 6,
Wherein the steps are substantially continuously repeated. ≪ Desc / Clms Page number 19 > The method according to claim 6,
Wherein the steps are repeated periodically. The method according to claim 6,
Characterized in that the steps are repeated when a pre-determined action requiring identification is performed by the user before being performed. The method according to claim 1,
Characterized in that the first camera is selected because a sufficiently large portion of the iris can be identified from the image so as to meet a minimum stringency threshold required to identify the user Providing system. A system for providing a substantially persistent biometric authentication for a user,
A headgear configured to be worn on a user's head;
A plurality of cameras mounted on the headgear, the plurality of cameras being directed from the different angles toward the eyes of the user when the headgear is worn to monitor movement of the eyes;
A processing unit operatively connected to a detector and an electronic display,
Analyze one or more images from the cameras to identify the iris of the eye;
Select images from various cameras providing desired views of the iris among the plurality of cameras;
Generate a composite iris code from the selected images;
Identify the user based at least in part on the composite iris code; And
And when the identity of the user is confirmed, permitting the user to perform one or more actions. A method of providing a substantially continuous biometric authentication for a user wearing a headgear, the headgear comprising a plurality of cameras directed from the different angles towards the user's eye,
Analyzing one or more images from the cameras to identify the iris of the eye;
Selecting images from various cameras providing desired views of the iris among the plurality of cameras;
Generating a composite iris code from the selected images;
Identifying the user based at least in part on the compound iris code; And
Allowing the user to perform one or more actions when the identity of the user is confirmed
The biometric authentication method comprising the steps of: 13. The method of claim 12,
Wherein the selected images provide images of the iris from a plurality of angles, and wherein generating the composite iris code comprises: selecting portions of the iris from the selected images and outputting the portions of the iris to the iris And combining the biometric information with a composite image for the biometric authentication. 13. The method of claim 12,
Further comprising rotating one or more of the selected images to align the images of the iris in the selected images on a common axis before combining the portions of the iris. How to provide authentication. 13. The method of claim 12,
After allowing the user to perform one or more actions,
Analyzing additional images from the cameras;
Generating a new composite iris code from the additional images;
Confirming the identity of the user based at least in part on the new compound iris code
The biometric authentication method further comprising: CLAIMS What is claimed is: 1. A method for providing a substantially continuous biometric authentication for a user wearing a headgear comprising a camera directed towards a user's eye,
Analyzing a plurality of images from the camera;
Generating a composite iris code based on the plurality of images
Identifying the user based at least in part on the compound iris code; And
Allowing the user to perform one or more actions when the identity of the user is confirmed
The biometric authentication method comprising the steps of: 17. The method of claim 16,
After allowing the user to perform one or more actions,
Analyzing additional images from the camera;
Generating a new composite iris code from the additional images;
Confirming the identity of the user based at least in part on the new compound iris code
The biometric authentication method further comprising: A system for providing a substantially persistent biometric authentication for a user,
A head gear configured to be worn on a user's head;
A detector mounted on the headgear, the detector being oriented towards the user's eye when the headgear is worn;
A display mounted on the headgear, the display being visible to the user when the headgear is worn; And
A processing unit operatively connected to the detector and the electronic display,
Analyze one or more images from the detector to identify features of the iris of the eye;
Identify the user based at least in part on the features of the iris;
Allow the user to perform one or more actions via the display when the identity of the user is confirmed; And
And changes the opacity of the display to preserve the security of information provided to the user on the display. 19. The method of claim 18,
Wherein the processing unit identifies one or more areas on the display where security information is provided and modifies the areas so that the areas become substantially opaque while the security information is provided System. 19. The method of claim 18,
Wherein if the identity of the user is associated with a predetermined threshold security authorization, the processing unit makes the entire display substantially opaque to protect subsequent security information on the display A system that provides biometric authentication. CLAIMS What is claimed is: 1. A method for providing a substantially continuous biometric authentication for a user wearing a headgear comprising a camera directed towards a user's eye,
Analyzing one or more images from the camera to identify features of the iris of the eye;
Identifying the user based at least in part on the features of the iris;
Authorizing the user to perform one or more actions when the identity of the user is confirmed;
Thereafter, performing further analysis to ensure that the user is still wearing the headgear
The biometric authentication method comprising the steps of: 22. The method of claim 21,
Said further analysis,
And analyzing one or more images from the camera to verify that the user's eye remains within the one or more images. 23. The method of claim 22,
Said further analysis,
And analyzing one or more images from the camera to verify that the user's eye remains active within the one or more images. 22. The method of claim 21,
Said further analysis,
And analyzing one or more images from the camera to identify features of the iris of the eye at a lower degree of severity than the initial authentication. 22. The method of claim 21,
After allowing the user to perform one or more actions,
Analyzing one or more additional images to identify features of the iris; And
Confirming the identity of the user based at least in part on the features of the iris
RTI ID = 0.0 > 1, < / RTI > 26. The method of claim 25,
a) substantially continuous; b) periodically; And c) a predetermined action is selected by the user requesting identification before being performed. ≪ Desc / Clms Page number 24 > 27. The method according to any one of claims 21 to 26,
Communicating the one or more images from the headgear, the steps of analyzing, identifying and authorizing are performed at a remote location from the headgear and a permission is communicated from the remote location to the headgear; And
Performing one or more authorized actions using the headgear
The biometric authentication method further comprising: 27. The method according to any one of claims 21 to 26,
Wherein at least one of the steps of analyzing, identifying and authorizing is performed by a processing unit on the headgear, the method comprising communicating with a processing unit at a remote location to perform the remainder of the analyzing, identifying and authorizing steps The biometric authentication method further comprising: 27. The method according to any one of claims 21 to 26,
Wherein the steps of analyzing, identifying and authorizing are performed by a processing unit on the headgear. CLAIMS 1. A method for providing a substantially continuous biometric authentication for a user wearing a headgear comprising a detector directed towards a user's eye,
Analyzing one or more images from the detector to identify features of the iris of the eye;
Identifying the user based at least in part on the features of the iris;
Allowing the user to perform one or more actions via the display when the identity of the user is confirmed; And
Changing the opacity of the display to preserve the security of information provided to the user on the display
The biometric authentication method comprising the steps of: A system for providing a substantially persistent biometric authentication for a user,
A head gear configured to be worn on a user's head;
A detector mounted on the headgear, the detector being oriented towards the user's eye when the headgear is worn;
A display mounted on the headgear, the display being visible to the user when the headgear is worn;
A sight shield mountable to the headgear, the field shield covering at least a portion of the display to prevent information provided on the display from being visible beyond the display;
A processor unit operatively connected to the detector and the electronic display,
Wherein the processing unit comprises:
Analyze one or more images from the detector to identify features of the iris of the eye;
Identify the user based at least in part on the features of the iris; And
And when the identity of the user is confirmed and when it is confirmed that the visual field breaker is mounted on the headgear, the user is allowed to access the information via the display. CLAIMS 1. A method for providing a substantially continuous biometric authentication for a user wearing a headgear, the headgear comprising a detector directed towards the user ' s eye,
Analyzing one or more images from the detector to identify features of the iris of the eye;
Identifying the user based at least in part on the features of the iris;
Detecting whether a visual field breaker covering at least a portion of the display is mounted on the headgear to prevent that information provided on the display can be seen beyond the display; And
When the identity of the user is confirmed and when it is confirmed that the visual field breaker is mounted on the headgear, allowing the user to access the information via the display
The biometric authentication method comprising the steps of: 33. The method of claim 32,
Detecting that the visual field breaker has been removed from the headgear; And
Prohibiting security information from being provided on the display when the visual field breaker is removed from the headgear
The biometric authentication method further comprising: 33. The method of claim 32,
Detecting that the visual field breaker has been removed from the headgear; And
Canceling the authority of the user when the visual field breaker is removed from the headgear
The biometric authentication method further comprising: A system for providing a substantially persistent biometric authentication for a user,
A head gear configured to be worn on a user's head;
A detector mounted on the headgear, the detector being oriented towards the user's eye when the headgear is worn; And
A processor unit operatively connected to the detector and the electronic display,
Wherein the processing unit comprises:
Analyze one or more images from the detector to identify features of the iris of the eye;
Identify the user based at least in part on the features of the iris; And
And when the identity of the user is confirmed, permitting the user to perform one or more actions. 36. The method of claim 35,
Further comprising a display operatively associated with the processing unit,
Wherein the processing unit is configured to permit the user to perform one or more actions via the display. 37. The method of claim 36,
Wherein the display is mounted on the headgear and the display can be viewed by the user when the headgear is worn. 37. The method of claim 36,
Wherein the display is remote from the headgear and the system further comprises a scene camera mounted on the headgear and wherein when the display is located in front of the user, Wherein the scene camera is oriented so that the scene camera is oriented in such a way that the scene camera is oriented. 36. The method of claim 35,
Wherein the processing unit comprises a processor mounted on the headgear. 36. The method of claim 35,
Further comprising a wireless transceiver on the headgear, and wherein the processing unit includes a processor remote from the headgear, the processor being wirelessly connected to the detector and the display via the wireless transceiver. / RTI > 36. The method of claim 35,
Wherein the detector comprises one or more cameras mounted on the headgear. 36. The method of claim 35,
Further comprising one or more illumination sources mounted on the headgear, wherein the illumination sources are directed towards the eye when the headgear is worn. 36. The method of claim 35,
The processing unit may also, after allowing the user to perform one or more actions,
Analyze one or more additional images to identify features of the iris; And
Wherein the verification of the identity of the user is performed at least once based on at least a portion of the features of the iris. 44. The method of claim 43,
The processing unit comprising:
a) substantially continuous; b) periodically; Or c) when a predetermined action that requires identification before being performed is selected by the user,
And to repeat the steps. CLAIMS 1. A method for providing a substantially continuous biometric authentication for a user wearing a headgear comprising a detector directed towards a user's eye,
Analyzing one or more images from the detector to identify features of the iris of the eye;
Identifying the user based at least in part on the features of the iris; And
Allowing the user to perform one or more actions via the display when the identity of the user is confirmed
The biometric authentication method comprising: 46. The method of claim 45,
After allowing the user to perform one or more actions,
Analyzing one or more additional images to identify features of the iris; And
Confirming the identity of the user based at least in part on the features of the iris
RTI ID = 0.0 > 1, < / RTI > 47. The method of claim 46,
a) substantially continuous; b) periodically; And c) a predetermined action is selected by the user requesting identification before being performed. ≪ Desc / Clms Page number 24 > 48. The method according to any one of claims 45 to 47,
Communicating the one or more images from the headgear, the steps of analyzing, identifying and authorizing are performed at a remote location from the headgear and a permission is communicated from the remote location to the headgear; And
Performing one or more authorized actions using the headgear
The biometric authentication method further comprising: 48. The method according to any one of claims 45 to 47,
Wherein at least one of the steps of analyzing, identifying and authorizing is performed by a processing unit on the headgear, the method comprising communicating with a processing unit at a remote location to perform the remainder of the analyzing, identifying and authorizing steps The biometric authentication method further comprising: 48. The method according to any one of claims 45 to 47,
Wherein the steps of analyzing, identifying and authorizing are performed by a processing unit on the headgear.

Images