WO2020041592A1

WO2020041592A1 - Evaluating user actions related to polymodal display information

Info

Publication number: WO2020041592A1
Application number: PCT/US2019/047709
Authority: WO
Inventors: Paul Atkinson; Jack DONNER
Original assignee: Chromera, Inc.
Priority date: 2018-08-22
Filing date: 2019-08-22
Publication date: 2020-02-27

Abstract

Devices, systems and methods for determining and evaluating users' actions in response to polymodal visible information, and taking actions according to the results of the evaluation. In one aspect of the invention, an information system has a device with a polymodal display for presenting information to a user with at least two modalities. The device also has a user action mechanism that accepts an action from the user. The action mechanism can be, for example, a mechanical device, an electrical or electronic device, a sensing device, a communication device or a position location device. The device also has an action detector that determines the actual action taken by the user, and then evaluation circuitry determines if the user performed an expected action in light of the information that was sent to the polymodal display.

Description

Evaluating User Actions Related to Polymodal Display

Information

Field of the Invention

[0001] The field of this invention is remote display devices with communication capabilities to a central server or network system.

Background

[0002] The highest and best outcome of many intelligent display devices often depends on users (humans or machines) acting (taking actions) according to, or in response to, changes in the circumstances or condition(s) of the device or the item or good to which the device is attached.

[0003] User compliance - actually taking the appropriate action (according to the current and circumstances and conditions), affects system/solution performance and integrity, and ultimately the outcomes of the goods, depends on the user

• Being aware of the need to act

• Knowing what action to take (given the circumstances and conditions)

• Having the ability to act (means, proximity, authorization etc.)

• Confidence and trust in the above, and...

• Having a reason to act (often with the foreknowledge that the action will produce beneficial outcomes, particularly those relating to the user).

[0004] The reason or motivation to act can be strengthened (and therefore compliance improved) with near frictionless access to, and a high level of confidence in, relevant actionable intel (awareness and knowledge) and the systems and methods that produce and present it. And further, by evaluating the users’ actions in the context of the actionable intel actually available to them.

[0005] User compliance is important to all the parties with stakes in the outcomes of the goods. And therefore, so is having confidence in the basis for user compliance: having access to trusted information (actionable intel) when/where it matters. Polymodal display devices - those having different display elements each having different operating modalities, is key to effectively presenting users with what they need to know. A simple four indicator polymodal display apparatus can present users with critical information - even if the power fails, and provide all the stakeholders in the good with a higher-level of confidence that users will comply with the information presented than they would have with conventional displays.

• Start-of-life (irreversible): passed inspection - fit-for-purpose

• Fail-safe (self-switching): the system has failed (e.g. has lost power)

• Warning (switchable): the good is about to expire, a device is about to fail

• End-of-life (permanent and non-switchable):“never use”

[0006] It should be noted, that having confidence in the medium (via a mix of modalities) builds trust not only in users and their actions, but also with remote 3rd parties, e.g. those participating in transactions related to the condition of goods (and therefore how users handle them), and who rely on information related to the presented information. The higher the level of confidence 3rd parties have in the presented information, the greater the confidence they’ll have in information they receive that directly or indirectly depends on it.

[0007] In short, feedback from, and accountability for, users’ actions responsive to visibly presented actionable intel, particularly that which is, or can be, polymodal (e.g. via a

polymorphic display), leads to higher, sustainable compliance and better outcomes. Visibly because it’s a near-frictionless, near universal method of accessing actionable intel. Polymodal, because it increases confidence in the presented intelligence (actionable intel) and actions, transactions and outcomes that depend on it.

[0008] There is therefore a need for systems and methods that can relate user actions (or lack thereof) with polymodal visible information, and in particular, with polymorphic displays.

And advantageously, to determine their causality. And in addition to raising compliance and item-level/item-specific outcomes, being able to improve/optimize system/solution design/ performance and aggregate outcomes: .

• Single, clusters, populations

• Local, remote

• Immediate, over time [0009] Summary of the Invention

[0010] Devices, systems and methods for determining and evaluating users’ actions in response to polymodal visible information, and taking actions according to the results of the evaluation. In one aspect of the invention, an information system has a device with a polymodal display for presenting information to a user with at least two modalities. The device also has a user action mechanism that accepts an action from the user. The action mechanism can be, for example, a mechanical device, an electrical or electronic device, a sensing device, a communication device or a position location device. The device also has an action detector that determines the actual action taken by the user, and then evaluation circuitry determines if the user performed an expected action in light of the information that was sent to the polymodal display.

[0011] Other embodiments of the invention have a plurality of remote devices, each with a polymodal display, which communicate with a server. In some cases the server may be a point of sale register, an RFID reader, a laptop, a cell phone, or a computer. These embodiments also have a user action mechanism, action detector, and evaluation circuty, but these may be located on the remote devices, on the server, or on both. Accordingly, the present invention provides a structure that can be deployed in a wide range of configurations.

[0012] The use of different modalities increases trust in the medium and thus the messaging, and user actions and remote transactions that depend on them. Feedback from, and

accountability for, users’ actions responsive to visibly presented actionable intel, particularly that which is polymodal leads to higher, sustainable compliance and better outcomes. Visibly because it’s a near-frictionless, near universal method of accessing actionable intel. Polymodal, because it increases confidence in the presented intelligence (actionable intel).

[0013] Brief Descriptions of the Drawings

[0014] Fig. l is a block diagram of an information system in accordance with the present invention.

[0015] Fig. 2 is a block diagram of a polymodal display in accordance with the present invention.

[0016] Fig. 3 is a block diagram of display devices and a server in accordance with the present invention. [0017] Fig. 4 is a block diagram of display devices and a server in accordance with the present invention.

[0018] Fig. 5 is a block diagram of display devices and a server in accordance with the present invention.

Detailed Description

[0019] Referring now to figure 1, an information system 100 is illustrated. In some

embodiments, information system 100 may be a device 102 that is configured as a product tag or label for an item or good (collectively referred to herein as goods), or integrated into a good’s packaging or container or into the good itself. It will be understood that the device 102 may be configured as many other types of products, such as forms, documents, cards, stickers, smartcards, healthcare products and consumables.. It will also be understood that the device 102 may be configured as, or incorporated into, standalone devices such as electronic shelf labels, advertising displays, electrical equipment, consumer electronics, healthcare devices, or mobile or Internet devices or appliances.

[0020] Device 102 has a polymodal display 105. Generally, a polymodal display is a display device or apparatus that has one or more pixels or indicators (display elements) operating in one mode, and one or more pixels or indicators (display elements) operating in a second mode. In figure 1, polymodal display 105 has display element 1 106 operating in a first mode, and display element 2 107 operating in a second mode. Generally, these modes relate to stability, switching, and transition characteristics of pixels. US patent application number 15/890,312, filed February 6, 2018 and entitled“Polymorphic Electro-Optic Displays,” which is incorporated herein by reference in its entirety, describes a type of polymodal display referred to as a polymorphic display. A polymorphic display is a unitary apparatus having multiple operable properties. Note that the terms pixel and indicator as used herein, should be understood to be single display elements.

[0021] Of particular interest are the operable properties, individually and in combination, of stability, switching and transition sequencing. The modes are briefly described below:

[0022] Stability

[0023] Mono-stable [0024] Mono-stable pixels have one, stable operating state (and corresponding optical state) and a second, volatile operating state (and corresponding optical state). Mono- stable pixels also have the switching properties of being reversible and self-switching. That is, they automatically, or“self’, switch from their volatile operating state to back their stable, first operating state when power to the pixel is terminated (or drops below a threshold level necessary to maintain the state).

[0025] A mono-stable pixels first, operating state is stable without power. When an electrical switching signal is applied to a mono-stable pixel, the pixel transitions from a stable, first operating state to a volatile, second, operating state. The volatile operating state is maintained as long as a maintenance signal is applied to the pixel. When the maintenance signal is terminated (for whatever reason) the pixel self-switches from the second operating state back to the first, stable operating state. Examples of mono-stable displays comprised of mono-stable pixels are common LCDs (liquid crystal displays), EPDs (electrophoretic and ECDs (electrochromic displays), and LEDs (light-emitting diodes) and OLEDs (organic light-emitting diodes).

[0026] Bi-stable

[0027] Bi-stable pixels have two stable operating states. Switching between the two stable operating states is accomplished with an electrical switching signal. Once switched, the operating state (and the corresponding optical state) persists when the power is terminated (without a maintenance signal). Bi-stable pixels may be reversible (e.g. EPDs, conventional ECDs, cholesteric, ferroelectric or zenithal bistable LCDs) or irreversible (as described in ETS patent No. 9,030,724 Flexible and Printable Electrooptic Devices). Conventional bi-stable pixels are electrically switched and are not self switching (but always switchable).

[0028] Multi-stable

[0029] A multi-stable pixel has three or more stable operating states. It may in addition also comprise one (or more) volatile states. For example, certain EPDs and Cholesteric LCDs may contain three stable states.

[0030] Permanent [0031] A permanent pixel is typically bi-stable and has the operable property that once switched from a first, stable state to a second, stable state it remains in the second state permanently - it can’t be switched back to its original, first state (or to a third state).

[0032] Switching

[0033] Electrically switchable

[0034] An electrically switchable pixel can be switched from one state to another state with the application of an electrical signal.

[0035] Self-switchable

[0036] A pixels having at least one volatile state may self-switch from the volatile to another state that is typically stable. The volatile operating state is maintained as long as a maintenance signal is applied to the pixel. When the maintenance signal is terminated (for whatever reason) the pixel self-switches from the volatile state to another state. As described above, if the display is mono-stable, the pixel will self-switch back to its original state. In certain multi-state pixels, the pixel may self-switch to a third state.

[0037] Non-switchable

[0038] A non-switchable pixel can be switched one time, from a first, stable state to a second, stable state. Once switched the pixel can’t be switched again. The pixel may however not be permanent and its (optical) state may fade over a relatively short period of time.

[0039] Transition Sequencing [0040] Reversible

[0041] A pixel may be switched from one state, back to its previous state. A mono- stable pixel is by definition reversible - it automatically reverses (self-switches ‘backwards’) from its volatile second state, to its first stable state when power is terminated. Most, but not all, bi-stable pixels are switchable/reversible. Certain multi- state pixels can be reversed between other combinations of states, e.g. from a second state to a third state, and the switched/reversed to the second state.

[0042] Irreversible

[0043] A pixel may not have the property of being reversible from one state to a previous one. Certain electrochromic pixels for example can be switched from a first state to a second state, but cannot be switched back (reversed) to their original, first state. Certain multi-state pixels however may be irreversible between a first state and a second state, but can transition‘forward’ from the second state to a third state.

[0044] Although the polymodal display device 105 may be in the form of a polymorphic display, it will be understood that other constructions may be used consistent with this disclosure. For example, the polymodal display 105 does not have to be a unitary device, but can be constructed such that display element 1 and display element 2 are separate and not on a single substrate. Further, it will be understood that although display element 1 and display element 2 are often closely spaced adjacent to each other, that display element 1 and display element 2 may be spaced apart.

[0045] Device 102 also has a power source (not shown) and a processor 109. It will be understood that the power source may be, for example, a battery, RF harvester or a photovoltaic cell. It will be understood that there are a large variety of devices that can power device 102. Device 102 also has a user action mechanism 111. The user action mechanism 111 is the mechanism by which a user takes an action(s) responsive to information displayed on the polymodal display device. The user action mechanism 111 generally is a mechanical device, an electrical or electronic device, a sensing device, a communication device or a location device that the user engages, and to which responses can be sensed, monitored, detected, or measured. In use, a user will be displayed information on the polymodal display 105. In some cases, responsive to the information displayed to the user, the user will interact with the user action mechanism 111.

[0046] Device 102 has an action detector 116 for detecting, directly or indirectly, responses to user actions from the user action mechanism 111. The action detector 116 is able to sense, monitor, detect, or measure responses to the user’s action. In this way, the device 102 is able to determine the action taken by user. The device 102 also has evaluation circuitry 122 which is used to evaluate the determined user action in light of the information written to the polymodal display 105, and in particular, what was written to on display element 1 and display element 2. Because the information displayed by display element 1 and display element 2 have different modalities, the evaluation circuitry 122 is able to evaluate the determined actions in context of the corresponding medium and message, and circumstances and conditions at the time the information was displayed and the time the user actions were taken (or not taken) has far more trusted information to work with then if display element 1 and display element 2 were of the same modality. In this way to the evaluation circuitry 122 is able to determine if the user performed an action consistent with the actionable intel, and further, did the user perform an expected action or perform an unexpected action.

[0047] By way of a simple example of a polymodal display 150, Fig. 2 illustrates a simple three-pixel electro-optic display 152 where each pixel (153, 154, 155) has two visible states: white and black. The display 152 is therefore capable of presenting up to eight different combinations (176-183) of visible actionable intel as illustrated at 175. Importantly, in this example the operable modalities (properties) of the pixels are different.

[0048] The first 153 and third 155 pixels are bi-stable and irreversible. In their unpowered, first state, they appear white. When power is applied to either pixel (153, 155) they switch to a second, black state. They remain in the black state even if the power is disrupted or terminated. And further, once switched to the black state, they cannot be switched back to their original white state. The second pixel 154 is volatile and self-switching. In its unpowered, first state the second pixel 154 is visibly white (like the first 153 and third 155 pixels). When power is applied, and for as long as power to the second pixel 154 is maintained, it switches to, and remains in, a second, black state. However, unlike the first 153 and third 155 pixels, if power to the second pixel 154 is disrupted or terminated (e.g. power, circuit, system failure, damage or tampering) the second pixel 154 automatically self-switches from its black state back to its original white state. For pixel 154 the process is repeatable.

[0049] In this example, the meaning of the visible actionable intel (the color of the pixels) depends on context. The context can be provided in a variety of ways, including supplemental visible information (e.g. printed information adjacent to the pixel). It can also be provided by the operable modalities of the pixels.

[0050] The operable properties of the three pixels (153, 154, 155) provide an added dimension to the actionable intelligence as illustrated below:

[0051] This added dimension or meaning, can affect user actions beyond that of typical displays.

[0052] In general, intelligent devices (e.g. device 102) and the systems and methods that incorporate them, comprise:

• Means of generating actionable intel

• Polymodal visible display elements/displays

• User action mechanism(s)

• User action detector(s)

• Means for evaluating determined user actions related to polymodal, visibly

presented actionable intel (and presented information related to determined user actions)

• Means for acting upon the evaluation [0053] It should be understood that the functions of an intelligent device may be configured/distributed across multiple devices.

[0054] Generating Actionable Intel

[0055] Exemplary means for generating actionable intel include some or all of the following:

• A processor and memory

^■ Configured with, and operating according to, loaded or embedded

instructions or“rules”

• One or more sources of power

• Internal/external sensors/monitors/detectors/receptors

^■ Temp, humidity, light, chemical, radiation, direction, motion, shock, vibration, pressure, mechanical, acoustic, optical, electrical, RF (or other EM) signals etc.

• Clocks/timers

• Actuators

• Wired/wireless (RF, optical, mechanical, electrical etc.) communications

• Geolocation (GPS, cellular tri angulation)

• Display circuitry (switching and/or maintenance signals)

• Secure Data Objects as described in ETS Patent applications 15/602,885 Agents and Systems for Rights Management

• Encryption algorithms, peer 2 peer and other communication protocols

[0056] Note that actionable intel may be generated according to individual, or combinations of, actual sensed or determined events (e.g. aggregate temperature exposure) or anticipated based on past events (e.g. a constant increase in vibration may be predicative that

motor/transmission/bearing/ gear/shaft etc. may be about to fail).

[0057] ETser Action Mechanism(s) (“action mechanism”)

[0058] A“user action mechanism” is any mechanism upon which a user (human or machine) can take an action (engage) that can be sensed/monitored/detected/measured -“determined”. And further, where the effect of the user action, the“determined user action”, can be evaluated in relationship to visible actionable intel presented by the intelligent device.

[0059] Note that a determinable user action may involve more than one step, and further may involve intermediate or complementary items such as a replacement part that is inserted into a socket, a fluid which is poured into a reservoir, a keypad by which an authorization code is entered, a keyboard by which inventory information is entered into a transaction system, an RFID reader or NFC enabled mobile phone is used to transmit information, etc. - in response to polymodal, visible actionable intelligence. Subsequent steps may also be driven by reactive display changes to initial action taken (e.g. moving item triggers display alert, replacing part triggers calibration alert, etc.)

[0060] The user action mechanism may be integral to the intelligent device (e.g. device 102), a good to which the intelligent device is attached or external to the intelligent device. If integral to an appropriately configured intelligent device, the evaluation can be internal to the intelligent device. If the user action mechanism is external to the intelligent device (e.g.,

proximate/local/remote), then the evaluation can take place at any appropriately configured device/location having access to the determined user action and related visible actionable intelligence. The determined user action and related visible actionable intelligence may be received by such a device and location, from the same, or different sources. Concurrently or temporally shifted.

[0061] The relationship between the determined user action and the visible actionable intel may be concurrent or temporally shifted. E.g., the catalyst for the determined user action may have been a lst visible actionable intel, but the action wasn’t taken until after the visible actionable intel was updated (e.g. a 2nd visible actionable intel presented).

[0062] Exemplary user action mechanisms include:

• Sockets, receptors, ejectors

• Latches, locks, snaps, zippers, seals

• Attachments, adhesives, screws, staples, bolts

• Enclosures, containers, doors, windows, gates

• Switches, contacts, buttons, connectors • Valves, handles

• Environmental or more generally, equipment control systems

• Motors, engines

• Keypads, keyboards, biometric devices

• Placement, movement, location (e.g. co-location, assembly), motion

^■ E.g. absolute or relative to another“good(s)” with separate intelligent device(s)

• Communication devices (wireless, wired) e.g., mobile phones, RFID readers, computer tablets

• Labels, packaging, forms, pick/check/workflow/QA/QC lists (e.g. variable printed information)

• Transaction systems (e.g. finance, pricing, payment, accounting, inventory, POS, reporting (regulatory, public, private), customer service, warranty, maintenance, shipping, track-and-trace, insurance, change of custody/security interests/title, blockchains (e.g. as described in ETS patent applications 15/228,270, Transaction Agents and Systems, and 15/602,885, Agents and Systems for Rights Management and which are incorporated herein in their entirety), etc.

• Turning on/off, enabling/disabling motors, equipment or electronic devices etc., remove or return to service (operation or location).

[0063] Note, user action mechanisms may be operable/controlled in a variety of ways, including directly mechanically controlled, or indirectly (e.g. electronically) controlled.

[0064] ETser Action Detector(s)

[0065] A user action detector, also referred to as an“action detector” is any means of detecting, sensing, monitoring or otherwise acquiring knowledge of user actions taken upon/with a user action mechanism. Exemplary action detectors include:

[0066] Sensors, e.g. temperature, humidity, moisture, liquid, gas, chemical, pressure, stress, proximity, torque, weight, shock, vibration, motion, accelerometers, location, proximity, magnetic, optical, light, acoustic, RF, electrical (e.g., capacitance, impedance, current, or voltage etc.), and more generally EM, etc. (machine or human perceptible ranges) • Communication devices (wired or wireless receivers)

• Electrical / optical circuits

• Location services (e.g. GPS, cellular)

• Transactions, transaction systems, information/databases, queries/updates etc.

• Corrupted memory, data, or protocol

[0067] Evaluation Circuitry

[0068] Means for evaluating the relationship(s) between visually presented actionable intel and determined user actions, and further, outcomes. The means / location where the evaluation is performed may be internal to the intelligent device, local/proximate or remote to the intelligent device, or distributed among multiple locations. In general, the evaluation means include:

• A processor, memory, EO (many of which may also serve as/with means of

generating actionable intel as described previously)

• Evaluation rules

^■ Distinct from the“conditional rules” that generate the visible actionable intel

^■ Evaluate determined actions in relationship to visible actionable intel and further, outcomes (intended/beneficial etc.)

^■ Determine / discover relationships between visible actionable intel and determined actions and outcomes

^■ Reflect the criteria by which the relationships are evaluated such as values, ranges, (absolute or relative), algorithms, strength, weights, confidence / uncertainty

^■ Typically pre-set (especially if the evaluation occurs in the intelligent device); Relationships are anticipated/intended thus the evaluation rules /criteria are (or can be) pre-set

^■ If external to the intelligent device, the rules/criteria can be preset or

dynamic

^■ If a cluster/multiple of intelligent devices (e.g. device 102) and

determinable actions (thus external), then using pattern recognition, using artificial intelligence other analytical tools, for determining/discovering/predicting relationship(s) (vs. having them

predetermined)

• Evaluation rules have access to:

^■ The visible actionable intel (and related information); variable + fixed (e.g., presented by a polymorphic display and printed information); and advantageously per the application, the determinants and dependents thereof (internal or external)

^■ The determined user actions (or lack thereof)

^■ And advantageously, factors affecting the relationship(s) e.g., information regarding the item/good, intelligent device or user(s)

^■ Current, past, changes in conditions / states

^■ Contextual information

• Internal/external conditions

• Location

• Other related, items, intelligent devices, users

• Custodians, owners... stakeholders

• Time

• Regulator/reporting requirements

• Market conditions

• History

• Available/altemative user actionable mechanisms

• Proximate / available users

^■ Outcomes information

^■ Rights, rules, relationships of the stakeholders in the item/good, label, user, actions or transactions

[0069] Referring now to figure 3, an information system 300 is disclosed. Information system 300 has a display device 302 similar to the display device 102, described with reference to figure 1. Accordingly, only differences will be described. As with display device 102, display device 302 has a polymodal display 305 having a display element 1 306 and a display element 2 307.

In contrast to display device 102, display device 302 does not have a user action mechanism 111, an action detector 116 or evaluation circuitry 122. Although not illustrated, display device 302 may include a processor, but due to its reduced computational requirements, the processor may be less powerful. Display device 302 is intended to be a remote display device, and therefore has communication circuit 325 for communicating over a network, such as a local area network, wide area network, or worldwide network such as the Internet. It will be understood that the communication circuit may be wired or wireless, such as a cellular, Wi-Fi or RFID wireless radio. The information system 300 includes multiple display devices similar to display device 302, such as display device 330 and display device 340. It will be understood that any number of display devices may be used. The display devices (302, 330, 340) communicate with a server 350.

[0070] The server 350 may be, for example, a point of sale register, an RFID reader, a laptop, a cell phone, or a computer. It will be understood that other apparatus may be used to function as server 350. The server 350 has a processor 310 for operating all server processes, and has communication circuit 326 for providing communication to the associated display devices (302, 330, 340). In contrast to information system 100, where the user action mechanism 111, action detector 116 an evaluation circuit 132 were located in a remote display device 102, in

information system 300 these functions are provided in server 350. It will be understood that although server 350 is shown as a single device, the functions within the server 350 may be distributed. Server 350 has user action mechanism 312, which may be for example, a keyboard input, and RFID reader input, a touchscreen, or a wireless input. It will be understood that a wide variety of electrical and physical mechanisms may be used as user action mechanism 312.

[0071] In use, a user would view information shown on polymodal display 305, and responsive to that information, take an action by engaging the user action mechanism 312 on server 350. Once the user has taken an action, the action detector 317 measures, monitors, detects, or confirms the responses/outcomes of the user action mechanism 312 and determines the action the user has taken. The server 350 then has evaluation circuit 323 for making a comparison between the action taken by the user, and the actionable intel written to the user’s polymodal display, such as polymodal display 305 on display device 302.

[0072] The evaluation circuitry 223 is able to use the actionable intel along with the actual action detected by the user, and determine if the user performed an action consistent with visual polymodal display 305. In this way, the evaluation circuitry 323 can determine if the user performed an expected action or performed an unexpected action.

[0073] Using the results of the evaluation, the server 350 is then able to take and perform further action. For example, the server 350 may transfer information to an accounting or financial system. In another example, the server 350 may make evaluation that the user is unauthorized or acting beyond their authority and therefore signal an audio, visual, or message alarm. In one example, the server 350 causes a new warning message to be displayed on a polymodal display. In another example, the server 350 may send a wireless communication warning that an unauthorized user is taking an unexpected action. Further actions are discussed below for acting upon the evaluation done by server 350.

[0074] Appropriate to the item/good, context or application the evaluation (and associated information) may be used to:

• Generate alerts, alarms, messages...

^■ Visible, audible, RF or other EM, haptic etc. communications

^■ Perceptible by humans or machines

• Initiate/affect/effect local/global transactions

• Adjust/update/reset internally, remotely

o Actionable intel

o Visual information

o Instructions

o Logic, conditional rules, variables

o Clock/timer

o Sampling rates / event storage rates

o Histories

• Update, repackage, replicate, create secure data objects etc. and

transmit/ di stribute

• Track actions in immutable stores such as blockchains

• Update chain of custody status [0075] It will be understood that server 350 has access to highly valuable trusted information regarding polymodal actionable intel and the actions users took in response to the displayed information. This information then forms the basis for performing actions with a level of confidence not permitted with prior information systems.

[0076] Referring now to figure 4, an information system 400 is illustrated. Information system 400 has a display device 402 similar to display device 102 described with reference to figure 1. Accordingly, only the differences will be discussed. As with display device 102, display device 402 has a polymodal display 405. Polymodal display 405 has display element 1 406 as well as display element 2 407. Display device 402 also has user action mechanism 411, which is similar to user action mechanism 111, so will not be described in detail. Display device 402 also has action detector 416, which is similar to action detector 116, so will not be described in detail. Display device 402 also has a communication circuit 425 for communicating to server 450. Communication circuit 425 is similar to communication circuit 225 described earlier, so will not be described in detail.

[0077] Information system 400 has several display devices, such as display device 402, 430, and 440. Each of these display devices has a communication circuit for communicating with a server 450. The server 450 has a processor 410 for performing functions for the server 450. The server also has a communication circuit 412 for providing wired or wireless communication to display devices, such as display devices 402, 430 and 440. The server 450 also has evaluation circuitry 423, which is similar to evaluation circuitry 223, so will not be described in detail. As with server 350, server 450 is able to take further actions based on the results of its evaluation.

[0078] As with server 350, it will be understood that server 450 has access to highly valuable trusted information regarding polymodal actionable intel and the actions users took in response to the displayed information. This information then forms the basis for performing actions with a level of confidence not permitted with prior information systems.

[0079] Referring now to figure 5, another information system 500 is disclosed. Information system 500 has a display device 502, which is similar to display device 102 described with reference to figure 1, accordingly, it will not be described in detail. Like display device 102, display device 502 has a polymodal display 505 having a display element 1 506 and a display element 2 507. The display device 402 also has user action mechanism 511, action detector 516, and evaluation circuitry 522. As with display device 402, display device 502 also has a communication circuit 525 for communicating to a server 540.

[0080] Information system 500 has multiple display devices, such as display devices 502, 530 and 540 which all communicate to a server 550. Server 550 may be, for example, a central computer system, a local area computer system, a point-of-sale system, a computer, a laptop, or cell phone. It will be understood that the server 550 may take alternative forms. The server 550 has a processor 510 for performing its functions, as well as a communication circuit 526 for providing wired or wireless communication to the multiple display devices, such as display devices 502, 530 and 540. Server 550 also has a user action mechanism 512. User action mechanism 512 is similar to user action mechanism 212 with reference to figure 2, so will not be described in detail. The user action mechanism 512 has an associated action detector 517, which is similar to action detector 217 described with reference to figure 2, so will not be described in detail. By having a user action mechanism 511 on the display device 502, and a user action mechanism 512 on server 550, a more complete understanding of a user action may be understood. Further, a user may take multiple actions within a short period of time. In one example, the user may take a local action by depressing a switch located on the display device 502, thereby implicating user action mechanism 511, and the user may also take an RFID interrogation from the display device 502, such that the server 550 detects that action using the user action mechanism 512. Thereby, by using both the remote user action mechanism 511 and the server user action mechanism 512, a complete understanding of a user action may be obtained, increasing the confidence that the user action is fully understood.

[0081] In a similar way, the remote display device 502 has evaluation circuitry 522, and the server 550 also has evaluation circuit 523. In this way user actions may be evaluated on the display device 502, and user actions may be further evaluated on server 550. It will be understood that the evaluated actions on server 550 may include a more detailed or sophisticated evaluation of the actions taken at display device 502, or may include actions accessible only to the server 550. As with server 450, server 550 is able to take further actions based on the results of its evaluation.

[0082] And as with server 450, it will be understood that server 550 has access to highly valuable trusted information regarding what messages or information a user should have been shown, and then actions that user took responsive to the displayed information. This information then forms the basis for performing actions with a level of confidence not permitted with prior information systems.

[0083] Exemplary Applications

[0084] Exemplary applications of information systems such as those previously described and comprising an intelligent device (e.g. device 102) configured appropriate to the use/application (e.g. labels, patches, packaging, tags, cards, discrete devices; single-use, multi-use etc.):

[0085] S ervi ce/Maintenance a. Aerospace / industrial/ military / transportation...

i. Maintenance/safety check of aircraft (or engines or... )

1. Status of intelligent device (ready to go, calibrated)

2. Status of test/assessment (e.g. active/functioning, or not)

a. Temp, vibration, moisture, stress...

3. Actions taken

a. Parts / fluids / seals / gaskets replaced)

b. Further test procedures, reporting

c. Enclosures closed

4. Intelligent device removed, reset for reuse

b. Hospital or construction equipment

i. Calibration

ii. Equipment tracking (secure areas, dangerous areas)

c. Automobile (consumer / service provider)

i. Periodic / usage service record

ii. Fluids, filters...

iii. Battery expiration date, times recharged

[0086] Workflow

a. “Steps” for given stage/item selected (among options) / presented

b. Steps from previous or current stage

i. Followed proper sequence

ii. Met‘success criteria’ (e.g. stage assembly/test) c. Movement from one secure area to another (or unsecure area)

i. Or from one environmental environment to another

ii. Or from one workstation (custodian) to another

d. Parts inventory updated

e. Project schedule updated

f. Update calibration data (SDO)

[0087] Custodial - supply chain

[0088] Inventory management

a. Environmental

[0089] Expiration

a. FIFE (first-in, first-expiration)

b. According monitored conditions a good is about to expire - has expired [0090] Security/tamper evidence/diversion

a. Visual alert/alarm when unauthorized action is taken

i. Opening, moving/relocating

ii. Tampering with monitoring system (e.g. shielding, disrupting power) iii. Subjecting to events/actions, intemal/extemal environments etc.

1. Outside‘normal’ operating ranges

2. Characteristic of attempts to defeat/interfere with intended/proper operation

[0091] Promotion

a. Electronic form of scratch-off ticket/card

i. Re-usable

ii. According to purchase amount

b. Loyalty card (earn / apply, spend via usage)

c. Gift card (programmable, stored value - via usage)

d. Stored value

i. Personalized

ii. Sync’d to on-line account

iii. Multi-location/participation

e. Update promotion data (pricing, products, marketing)

[0092] Healthcare a. Expiry date label (exemplary of safety)

b. Patient care

i. E.g. hospital/bedside history (electronic chart)

1. Medications

2. Checks / examinations

ii. Patches

iii. Status of intelligent device (new/use, working, used/don’t use)

iv. Status of patient / monitored condition results

[0093] Agriculture

[0094] Animals

a. Vaccination / medical record

b. ID, owner

c. Monitored conditions (e.g. health)

[0095] Global transactions

a. Liability/security interests

b. Financial

i. Pricing

ii. Financing

iii. Settlement

c. Insurance

d. Regulatory reporting

e. Warranty

[0096] Customer assurance

[0097] Advantageously, the information systems described herein may be configured to:

[0098] Detect, map, and evaluate multiple (e.g. a series or sequence of) user actions in relation to a single, or series of related, polymodal visual presentations of actionable intel.

[0099] Detect and evaluate the actions of multiple users in relation to polymodal visual presentations of actionable intel. E.g. a chain of custody, workflow... [0100] Distinguish between authorized and unauthorized users or otherwise‘bad actors’ (e.g. the intelligent device, or an external user action means, is equipped to authenticate users, authorize access or enable actions according to the actionable intel.

[0101] Distinguish‘good’ actors from‘bad actors’ (e.g. by detecting and evaluating unauthorized, inappropriate or otherwise‘outside the norm’ actions in relation to the polymodal visual presentation of actionable intel.

[0102] Detect’ and evaluate the absence of a user action or actions (‘null actions’) in relation to the polymodal visually presented actionable intel.

[0103] Detect and evaluate incorrect, incomplete or unintended user actions in relation to the polymodal visually presented actionable intel (particularly in relation to the presence (or absence) of users (and their authorization or capabilities) and user action mechanisms.

[0104] While particular preferred and alternative embodiments of the present intention have been disclosed, it will be appreciated that many various modifications and extensions of the above described technology may be implemented using the teaching of this invention. All such modifications and extensions are intended to be included within the true spirit and scope of the appended claims.

Claims

CLAIMS What is claimed is:

1. An information display system, comprising:

a polymodal display apparatus constructed to have at least two display elements, with the first element having an operating modality different from the second element;

a processor causing a message to be written to the polymodal display apparatus;

a user action mechanism for accepting an action from a user;

an action detector coupled to the action mechanism and constructed to determine what action was taken by the user; and

evaluation circuitry for evaluating if the determined action is an expected action by comparing the determined action to the message written to the display at a time prior to the user taking the action.

2. The information display system according to claim 1, wherein each of the operating modalities is selected from the group consisting of (1) electrically irreversible, (2) permenanant, (3) permanent and irreversible, and (4) volatile and self switching.

3. The information display system according to claim 1, wherein the first element and the second element have a (1) different stability, (2) different switchability, or (3) different transition sequencing.

4. The information display system according to claim 1, wherein the polymodal display apparatus comprises a polymorphic display.

5. The information system according to claim 1, wherein the polymodal display apparatus, user action mechanism, the processor, the action detector, and the evaluation circuitry are all integrated into the same apparatus.

6. The information display system according to claim 1, wherein the polymodal display apparatus is integrated into a label, document, card, sticker, smart card, or healthcare patch, packaging or container.

7. The information display system according to claim 1, wherein the polymodal display apparatus is integrated into an electrical or electronic product or equipment, electronic shelf label, advertising display, mobile phone or mobile electronic device, or internet appliance.

8. The information display system according to claim 1, further including a sensor coupled to the processor.

9. The information display system according to claim 8, wherein the sensor is a sensor for detecting temperature, humidity, moisture, liquids, gases, weight, light, chemicals, radiation, direction, motion, acceleration, shock, vibration, pressure, torque, capacitance, impedance, current voltage, magnetism, sound, wired or wireless communications or signals, location or proximity.

10. The information display system according to claim 1, wherein the user action mechanism is integral to the polymodal display apparatus.

11. The information display system according to claim 1, wherein the user action mechanism is a mechanical device, an electrical or electronic device, a sensing device, a communication device or a location device that the user engages.

12. The information display system according to claim 1, wherein the user action mechanism is wherein the user action mechanism is a communication system or a transaction system that the user engages.

13. The information display system according to claim 1, wherein the action detector receives a signal from: a sensor, an environmental sensor, an electrical or electronic device, a communication device, a location service, or a transaction system.

14. The information display system according to claim 1, further including a clock, a timer, an elapsed timer, or an absolute timer.

15. The information display system according to claim 1, further including determination circuitry, which responsive to evaluating that the action taken by the user was not expected, determines that the user in an unauthorized user.

16. The information display system according to claim 1, further including determination circuitry, which responsive to evaluating that the action taken by the user was not expected, determines that the display apparatus did not function correctly.

17. The information display system according to claim 1, further including determination circuitry, which responsive to evaluating that the action taken by the user was expected, authorizing a transaction.

18. The information display system according to claim 1, further including determination circuitry, which responsive to evaluating that the action taken by the user was not expected, (1) writing a new message to the display apparatus, (2) setting an audible or visual alarm, (3) using communication circuitry to send a wireless communication; or (4) using communication circuitry to send an RF signal.

19. An information system, comprising:

a plurality of display devices, each having a polymodal display apparatus;

wherein each respective polymodal display apparatus has at least two display elements, with the first element having an operating display modality different from the second element, and a processor constructed to write messages to the display apparatus;

a communication circuit in each respective display device for transmitting information related to messages written to the display devices;

a server, comprising;

a communication circuit for receiving the information related to messages written to the display devices transmitted by the display devices; a user action mechanism for accepting an action from a user regarding each respective display device;

an action detector coupled to the action mechanism and constructed to determine what action was taken by the user at each respective display device; and

evaluation circuitry for evaluating if the determined action is an expected action by comparing the determined action to information related to the message written to the display at a time prior to the user taking the action.

20. The information system according to claim 19, wherein each of the operating modalities is selected from the group consisting of (1) electrically irreversible, (2) permanent , (3) permanent and irreversible, and (4) volatile and self switching.

21. The information system according to claim 19, wherein the first element and the second element have a (1) different stability, (2) different switchability, or (3) different transition sequencing.

22. The information system according to claim 19, wherein the polymodal display apparatus comprises a polymorphic display.

23. The information system according to claim 19, wherein each display device is integrated into a label, document, card, sticker, smart card, or healthcare patch, packaging or container.

24. The information system according to claim 19, wherein each display device is integrated into an electrical or electronic product or equipment, electronic shelf label, advertising display, mobile phone or mobile electronic device, or internet appliance.

25. The information system according to claim 19, further including a sensor coupled to the processor.

26. The information system according to claim 25, wherein the sensor is a sensor for detecting temperature, humidity, moisture, liquids, gases, weight, light, chemicals, radiation, direction, motion, acceleration, shock, vibration, pressure, torque, capacitance, impedance, current voltage, magnetism, sound, wired or wireless communications or signals, location or proximity.

27. The information system according to claim 19, wherein the user action mechanism is a mechanical device, an electrical or electronic device, a sensing device, a communication device or a location device that the user engages.

28. The information system according to claim 19, wherein the user action mechanism is a communication system or a transaction system that the user engages.

29. The information system according to claim 19, wherein the action detector receives a signal from: a sensor, an environmental sensor, an electrical or electronic device, a

communication device, a location service, or a transaction system.

30. The information system according to claim 19, wherein the display devices further each include a clock, a timer, an elapsed timer, or an absolute timer.

31. The information system according to claim 19, wherein the server further includes determination circuitry, which responsive to evaluating that the action taken by the user was not expected, determines that the user in an unauthorized user.

32. The information system according to claim 19, wherein the server further includes determination circuitry, which responsive to evaluating that the action taken by the user was not expected, determines that the display apparatus did not function correctly.

33. The information system according to claim 19, wherein the server further includes determination circuitry, which responsive to evaluating that the action taken by the user was expected, authorizing a transaction.

34. The information display system according to claim 1, further including determination circuitry, which responsive to evaluating that the action taken by the user was not expected, (1) writing a new message to the display apparatus, (2) setting an audible or visual alarm, (3) using the communication circuitry to send a wireless communication; or (4) using the communication circuitry to send an RF signal.

35. An information system comprising

a plurality of display devices, each having a polymodal display apparatus and a communication circuit;

wherein each respective polymodal display apparatus has at least two display elements, with the first element having an operating display modality different from the second element, and a processor constructed to write messages to the display;

a user action mechanism for accepting an action from a user;

an action detector coupled to the user action mechanism and constructed to determine what action was taken by the user; and

a server, comprising;

a communication circuit for receiving information related to write- messages from the display device and information related to the determined action; and

evaluation circuitry for evaluating if the determined action is an expected action by comparing the determined action to the write message information received from the display device.

36. The information system according to claim 35, wherein each of the operating modalities is selected from the group consisting of (1) electrically irreversible, (2) permanent, (3) permanent and irreversible, and (4) volatile and self switching.

37. The information system according to claim 35, wherein the first element and the second element have a (1) different stability, (2) different switchability, or (3) different transition sequencing.

38. The information system according to claim 35, wherein the polymodal display apparatus comprises a polymorphic display.

39. The information system according to claim 35, wherein each display device is integrated into a label, document, card, sticker, smart card, or healthcare patch, packaging or container.

40. The information system according to claim 35, wherein each display device is integrated into an electrical or electronic product or equipment, electronic shelf label, advertising display, mobile phone or mobile electronic device, or internet appliance.

41. The information system according to claim 35, further including a sensor coupled to the processor.

42. The information system according to claim 41, wherein the sensor is a sensor for detecting temperature, humidity, moisture, liquids, gases, weight, light, chemicals, radiation, direction, motion, acceleration, shock, vibration, pressure, torque, capacitance, impedance, current voltage, magnetism, sound, wired or wireless communications or signals, location or proximity.

43. The information system according to claim 35, wherein the user action mechanism is integral to the polymodal display apparatus.

44. The information system according to claim 35, wherein the user action mechanism is a mechanical device, an electrical or electronic device, a sensing device, a communication device or a location device that the user engages.

45. The information system according to claim 35, wherein the user action mechanism is a communication system or a transaction system that the user engages.

46. The information system according to claim 35, wherein the action detector receives a signal from: a sensor, an environmental sensor, an electrical or electronic device, a

communication device, a location service, or a transaction system.

47. The information system according to claim 35, wherein the display devices further each include a clock, a timer, an elapsed timer, or an absolute timer.

48. The information system according to claim 35, wherein the server further includes including determination circuitry, which responsive to evaluating that the action taken by the user was not expected, determines that the user in an unauthorized user.

49. The information system according to claim 35, wherein the server further includes including determination circuitry, which responsive to evaluating that the action taken by the user was not expected, determines that the display apparatus did not function correctly.

50. The information system according to claim 35, wherein the server further includes including determination circuitry, which responsive to evaluating that the action taken by the user was expected, authorizing a transaction.

51. The information display system according to claim 1, further including determination circuitry, which responsive to evaluating that the action taken by the user was not expected, (1) writing a new message to the display apparatus, (2) setting an audible or visual alarm, (3) using communication circuitry to send a wireless communication; or (4) using communication circuitry to send an RF signal.