US20220261812A1

US20220261812A1 - Interfacing with a point of sale system from a computer vision system

Info

Publication number: US20220261812A1
Application number: US17/592,287
Authority: US
Inventors: William Glaser; Brian Van Osdol
Original assignee: Grabango Co
Current assignee: Grabango Co
Priority date: 2017-06-21
Filing date: 2022-02-03
Publication date: 2022-08-18
Also published as: AU2023202100A1; US20180373928A1; US11748465B2; AU2018289552B2; AU2018289552A1; US20240134950A1; US20200394636A1; IL271528B1; WO2018237210A8; BR112019027120A2; IL271528A; US11288650B2; US20220237621A1; EP3635623A4; WO2018237210A1; EP3635623A1; US20190005479A1; US10740742B2

Abstract

A system and method for interfacing with a point of sale system from a computer vision system that includes, through a CV monitoring system, detecting a computer vision modeled person and setting a user-record in association with a modeled person, the CV monitoring system being part of a computing platform; modeling interactions of the modeled person as modeled data associated with the user-record; detecting the modeled person in proximity to a computing device in the environment and thereby establishing an association of the computing device with the modeled person; and after establishing an association of the computing device with the modeled person, transmitting at least a part of the modeled data from the computing platform to the computing device.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

This Application is a Continuation Application of U.S. patent application Ser. No. 16/015,181, filed on 21 Jun. 2018, which claims the benefit of U.S. Provisional Application No. 62/523,183, filed on 21 Jun. 2017, both of which are incorporated in their entirety by this reference.

TECHNICAL FIELD

This invention relates generally to the field of computer vision, and more specifically to a new and useful system and method for personalized interactivity with a shared computer vision system.

BACKGROUND

Computer vision and sensor fusion have seen some early applications to commerce and payment. Many of these applications though are limited in nature. Many applications that expand partially beyond basic functionality rely on a fully integrated system and thus are cost and logistically prohibitive for such systems to be used in existing places of business. Thus, there is a need in the computer vision field to create a new and useful system and method for interfacing with a computing device from a computer vision system. This invention provides such a new and useful system and method.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 is a schematic representation of an exemplary implementation of a system of a preferred embodiment;

FIGS. 2-4 are schematic representations of exemplary forms of associations;

FIG. 5 is a schematic representation of an exemplary implementation of the system applied across multiple humans in an environment;

FIG. 6 is a flowchart representation of a method of a preferred embodiment;

FIG. 7 is a flowchart representation of a variations of establishing associations;

FIGS. 8 and 9 are schematic representations of exemplary variations of establishing an association with a device instance;

FIGS. 10A-10F are schematic representations of exemplary variations of establishing an association during a pairing process of a device instance;

FIG. 11 is a schematic representation of a sequence of association mechanisms;

FIGS. 12-14 are schematic representations of coordinating with a secondary device.

DESCRIPTION OF THE EMBODIMENTS

The following description of the embodiments of the invention is not intended to limit the invention to these embodiments but rather to enable a person skilled in the art to make and use this invention.

1. Overview

A system and method for interfacing with a computing device from a computer vision system functions to enable computing devices in an environment to pair, synchronize, or otherwise associate with a computer vision (CV) detected and monitored person or object.
The system and method are preferably used in coordinating an interactive experience that is in part based on a computer vision (CV) model of the environment and one or more computing devices in that environment. The computer vision model may model people, objects, the environment and interactions/events in that environment, and/or other aspects. The system could similarly be applied to other sensor-driven interaction experiences.
In a preferred variation, the system and method are used to synchronize state information associated with a CV monitored person to a computing device as the person comes into proximity of the computing device. This can be used to transfer data from the computing platform of the CV system to the computing device. This may also be used to relay communication from the computing device back to the computing platform and associated or used with the CV-based data modeling in the computing platform. While there are a wide variety of applications, one preferred application is in the interfacing of a checkout station (i.e., a point of sale system) from a computer vision system. In particular, checkout list information collected by the CV monitoring system on behalf of a customer can be automatically transferred to a checkout station when it is the customer's turn to checkout, thereby expediting and speeding up the checkout process. The process could similarly be used in transferring payment information for performing a transaction. The system and method may be applied to other suitable applications as described herein, but in general, the system and method are described as they would apply to commerce and in particular the checkout process with a checkout station.
Integrating with computing device infrastructure of an environment can enable CV-based monitoring or other forms of sensor-based monitoring to drive or facilitate interaction experiences involving these computing devices. These computing devices could be checkout stations (as discussed above), informational kiosks, automated teller machines of a bank, vending machines, and/or any suitable type of computing device or machine. In some variations, interfacing with the computing device infrastructure can involve a physical interface device which can act as a proxy for digitally interfacing. In other variations, a programmatic interface such as a network accessible application programming interface to remotely interface with a computing device (e.g., a POS station). This could be wirelessly or in a wired connection.
The system and method can be used in large scale CV-driven applications where CV-based modeling of people, objects, the environment, and interactions in that environment are used across independent and personalized digital experiences of different participants. For example, these CV-driven applications can be across large environments such as a grocery store, a mall, or any suitable environment that would depend on multiple camera imaging. Furthermore, such CV-driven interactivity may be enabled ad-hoc for people in the environment.
The system may apply a process of establishing associations between CV modeled people with other elements.
The association(s) established by the system and method can be between a CV-person, a user-record and or a user-account, and then preferably with some other associative element. The additional associative element can be a human (wherein direct or indirect properties of a person can be used to identify a person) and/or a computing device (e.g., a physical computing device, and/or some application instance on a computing device). These associations can additionally be scoped to a particular environment or space and period of time. The system and method may additionally support operating with different types of association “topologies”.
A CV-person (i.e., a computer vision modeled person) as used herein characterizes a detected and optionally tracked physical person modeled and represented by a CV monitoring system. A CV-person may, or may not, have any personally identifying information or person-associated metadata. The CV-person will generally represent the presence of a person in an environment and optionally the location, actions, activity, CV-derived properties, and/or other information based on image processing. For example, a CV-person could have a tracked location or position, body properties (e.g., pose, clothing, or other visual properties), and/or other properties. CV-detected events, gestures or states may additionally be detected in relationship to the CV-person and form part of the data model/representation of that physical person. Here a CV monitoring system characterizes an imaging system and/or processing system employing computer vision to facilitate the detection and tracking of objects, detecting actions, and performing other CV-based operations. Herein, the system and method is described as it applies to CV-based modeling of people, but a CV monitoring system may additionally or alternatively model other entities or objects such as animals, vehicles, automated devices/robots, or other suitable objects.
A user-record as used herein characterizes a computer record that is associated with a single human or being. A user-record can preferably be associated with a CV-person to uniquely identify, track, and record state of the CV-person. A user-record will preferably include various properties. Properties of a user-record may include personal identifiable information, but it may alternatively not include any identifiable information. A user-record in some implementations may also include payment information, user settings, user history, privilege settings/permissions, and/or other information. Properties of a user-record may initially use default settings that are used as placeholders until these properties can be updated. Properties may be updated as a result of association with other elements or through CV-based monitoring of the CV-person.
A user-record can additionally be used in modeling the state of an entity within a system, which will generally be directing in part a digital interactive experience. For example, in a variation of the system and method applied to automatic checkout powered by CV, the user-record may be the record used to record the items selected for purchase forming some.
A user-record can be an internal model of a human. As an internal model of a human, the user-record could be an ephemeral user-record that is preserved for essentially over the duration that the CV-person is detected in the environment. For example, a user-record can be created and updated for the duration that a shopper is present in a store. Additionally, a user-record may bridge the CV-modeling of a person across non-continuous CV monitoring of a person. For example, a first instance of a CV-person monitored in one location at one particular time period and a second instance of a CV person monitored at one location and at second time period (e.g., there may be a gap in video capture as the person enters an unmonitored area) could be associated with the same user-record. A user-record could additionally be persistent over multiple visits or system sessions, wherein a user-record may be reestablished for a human over multiple visits to the environment. As an internal model of a human, a user may not explicitly create nor have direct access to manage the user-record. In one variation, the user-record can be, include, or reference a user-account, where at least a part of the user-record may be user managed. In some implementations, the system and method may operate with a unified concept of user-record and user-account. In such an implementation, the user-record can be a user-account with user managed authentication credentials. In other implementations, verified user-accounts and ephemeral/anonymous user accounts could be used in a similar manner. Other suitable user modeling could similarly be applied.
A user-account as used herein characterizes an account in the system that can be accessed and manipulated by a user. A user-account will generally be accompanied with some authorization process such as a username and password or other authentication credentials or process. The user-account may reference a user-account of an outside system such as a social media platform. In the exemplary use case of the system applied to an automatic checkout, a user-account may be used to enable a user to register an application instance, edit payment methods, change account settings, view shopping history, edit shopping lists, and/or perform other actions. These actions can be synchronized with the user-account and can be performed outside of a CV-system, but can preferably be leveraged by a CV-system when the system and method associates a CV-person and a user-account. In some variations, a user-account could be implemented as an optional associative element for humans in the environment. CV-driven applications may handle humans with only ephemeral user-records as well as humans with user-accounts.
In some variations, a user-record can optionally include or reference (through some link or association) a user-account. In the situations where the system and method can optionally use a user-account, some humans will have only a user-record used in internally modeling state and referencing CV-related data. These humans may not have a user-account. In this exemplary situation, some other humans will have a user-record that is associated with or is their user-account. In other implementations, only humans with explicitly created user-accounts may be stored. Supporting both ephemeral/anonymous user-records and user-accounts can have the benefit of enabling ad-hoc association of a new user-account with an existing user-record. For example, a user that creates a user account in the middle of a shopping trip can gain access to their CV-based modeling (e.g., a virtual cart for automatic or expedited checkout) that was tracked through a previously anonymous user-record.
As will be appreciated by one knowledgeable in the art, the exact data model architecture of such records can take many forms. Herein, a user-record will generally be used to reference the generic user-record, which may or may not have association to a user-account with authentication credentials, user-account will generally be used in situations where there is a user-created record that the user is able to authenticate into unless otherwise specified.
A human as used herein characterizes a human being with a unique identity. A human is sometimes referred to as the user, customer, worker, or other human label where appropriate. In the exemplary use case of the system applied to automatic checkout, a human can be someone who is a shopper, a worker in the store, or is otherwise present in the environment. A human could similarly generalize to other living or non-living entities such as farm animals, pets, vehicles, robots, and the like. Herein, we will primarily reference humans but one knowledgeable in the art would appreciate that the system and method are not limited to use with humans.
A device instance as used herein characterizes a unique instantiation of a stateful device present in the environment. A device instance in many cases is a computing device such as smart phone, a wearable computer, a tablet, a personal computer, or a computing kiosk/station. A device instance is preferably uniquely identifiable. A first type of device instance (sometimes referred to simply as a computing device) is the one to which the computing platform is interfacing with. As such the computing device instance could be computing devices accessible within the environment such as a point of sale (POS) payment kiosk, a help kiosk, or any suitable type of computing device.
In some cases, the system and method may additionally use other types of device instances in establishing an association and/or collecting or delivering data. In some variations, a device instance may be an application instance on a device such as an application installed by a user. Here, instance is used to distinguish a unique installation and instantiation of an application on a particular device. Continuing with the use case of the system applied to automatic checkout, the application instance can be an application of the automatic checkout service that a shopper can use to view their currently selected items, to complete checkout, to modify settings, and/or perform other tasks. In some instances, the system and method can facilitate synchronizing the state of the application instance with the user-record.
An association as used herein is some variation of symbolic linking or connecting of at least two associative elements selected from the set of a CV-person, a user-record, user-account, a device instance, and/or a human. More preferably, the association is a three-element association of a CV-person and a user-record with at least one of a user-account, a device instance, and or a human. A CV-person is preferably detected and then matched to a corresponding identifying associative element. The identifying associative element can be a device instance, human, or an identifier of a user-account. The identifying associative element may then be linked to a user-record.
An association with computing device infrastructure preferably includes an association with at least a user-record to which CV-based data modeling is connected. For example, a checkout list for a particular customer can be stored in association with the user-record. That checkout list (or other user-record related data) can be transferred in part or whole to a computing device.
An association that includes a human association as shown in FIG. 2 can provide positive confirmation of that human. An association with a human can be through biometric detection of the human through the CV monitoring system. A human association could also be indirectly established through biometric identification through a device/application. A human association could additionally use other identifying properties of a human such as interaction patterns (e.g., shopping patterns), clothing patterns, patterns in accompanying humans, and/or other properties.
An association that includes a device instance association as shown in FIG. 3 can establish state synchronization with the device and/or application. Interactions with an app can influence the CV monitoring system and similarly detected information from the CV monitoring system can be pushed to the device or application instance.
An association that includes a user-account as shown in FIG. 4 can establish state synchronization with the user-account to leverage stored user associated data and settings. Stored settings of an account can influence the CV monitoring system and similarly detected information from the CV monitoring system can be used to update the user-account.
An association with a user-record preferably provides a level of indirection between the CV-person and identifying associative elements so that CV modeling updates can be resilient to delays in establishing an identifying association or changes in associations.
An association is preferably maintained during the duration of the CV-person (e.g., duration of a shopper shopping in a store). The association is preferably maintained across multiple visits to multiple stores—the association will generally be reestablished for each visit. The system and method can facilitate the establishment and/or reestablishment of the association. An association can additionally be expanded to include other associative elements. Elements of an association could however undergo permanent disassociation (e.g., removing a device instance) or temporary disassociation (e.g., disassociating a user-account for anonymous shopping during a single visit).
An association topology generally describes a collection of associations to different associative elements as shown in FIG. 1. Different topologies may lend themselves to different capabilities. Different interaction experiences may also depend on or benefit from particular types of topologies. The association topology may additionally involve the type of association that was established. The type of association can be linked to different levels of identification confidence, privileges, or other aspects. For example, identification of a computing device using one technique may be treated differently from identification through another technique. Additionally, associations can be reinforced or occur at different times and/or locations to affirm the correct association. Furthermore, understanding of associations can be applied longitudinally with the monitoring of a person, which is to say, an association established at one point in time could be applied to future occurrences or retroactively applied, even altering previous state based on the new association information. In viewing the interaction experience as being based around an association topology, the system and method could additionally be extended to providing shared digital experiences and interactions for groups. In this way, multiple humans and/or multiple devices may have a shared digital experience as shown in FIG. 5. For example, a family shopping together would be represented as multiple CV-people, but those modeled interactions may all be linked back to a shared family user-account.
The environment as used herein characterizes the site where a CV monitoring system is installed and operational. The system and method can be made to work for a wide variety of environments. In a preferred implementation, the environment is a shopping environment such as a grocery store, convenience store, micro-commerce & unstaffed store, bulk-item store, pharmacy, bookstore, warehouse, mall, market, and/or any suitable environment that promotes commerce or exchange of goods or services. In alternate use cases, the environment can include a household, an office setting, a school, an airport, a public/city space, a gym, and/or any suitable location. The environment can be a locally contained environment but may alternatively be a distributed system with wide coverage.
The system and method may be used for any suitable video monitoring application. In particular, the system and method is for CV-based computing applications, which may include automated self-checkout, inventory tracking, security surveillance, environmental event detection and tracking, and/or other suitable applications.
Herein, automatic self-checkout is used as the main exemplary application of the system and method, but any suitable application may be used. Automatic self-checkout is primarily characterized by a system or method that generates or maintains a virtual cart during the shopping process of a customer (or group of customers) with the objective of knowing the possessed items when a customer leaves a store. An automatic checkout system may additionally automatically charge an account of a customer for the total of a virtual cart (e.g., predicted list of items selected for purchase). The automatic checkout system could alternatively be used to expedite entry of items for payment. The system and method as described herein can be used to synchronize a CV-person and a resulting generated virtual cart (as detected in part or whole through the CV monitoring system) with a user-record, user-account, device instance, and/or a human. This could then be used to enable automatically charging of a stored payment method, authorizing purchase of restricted items (e.g., alcoholic beverages and the like), or execute other tasks.
The system and method may alternatively be used to account for the removal of a good by a customer such as in a library, a rental store, a warehouse, or any suitable item storage facility and update an associative element appropriately. The system and method may alternatively be used to permit or restrict access to locations, to charge for such access, to enable authentication with shared computing devices, or for any use case. The system can be made to work for a wide variety of shopping environments such as grocery stores, convenience stores, micro-commerce & unstaffed stores, bulk-item stores, pharmacies, bookstores, warehouses, malls, markets, and/or any suitable environment that promotes commerce or exchange of goods or services. The system and method can also be used in other situations or in other environments.
The system is primarily described as employing a network of multiple video monitoring nodes. The system could additionally be applied to a single camera and optionally audio monitoring and may additionally include other forms of sensing that may be used independently or in combination with video monitoring. These might include lidar, sonar, infrared, ultraviolet, radio frequency, capacitance, magnetic, scales, pressure, volume, climate sensing, and the like.
As one potential use, the system and method can be applied for synchronizing user-account data or application state of CV-based monitoring of a CV-person with encountered or used computing devices. This may even be performed independent of the user having ever used or downloaded an application for the checkout service before or creating a user-account for such a service. This synchronization of state between a CV monitoring system and a device can be coordinated with a human's proximity to a device or usage of the device. In one example this may be used in coordinating communication of user-record data such as a virtual cart with store installed computing devices like a checkout station. For example, a virtual cart generated for a CV-person could be communicated to a POS system when the CV-person becomes associated with the POS system after the CV-person is within some spatial proximity of the POS system. In another example, the system and method can be used to synchronize information with a publicly accessible kiosk when the human walks up and starts interacting with the kiosk. From the perspective of the human, the kiosk would seemingly automatically recognize them and personalize the interactions.
As another potential benefit, the system and method can provide a seamless user experience. Data, information, and state collected using a CV-based monitoring system and/or other sensing systems can be automatically synchronized with a computing device when the person comes within a defined proximity of the computing device.
Related to this enhanced user experience, the system and method can enhance efficiency for many systems. In applying the system and method to commerce, the checkout process can be made significantly faster by automatically feeding a checkout list to a checkout station.
As one potential benefit, the system and method can be integrated with existing systems of the environment. In a store for example, the system and method can be integrated with the point-of-sale system used by the store. A point and sale system is often deeply integrated into the operations of a store. Integrating with an existing point of sale system may offer a number of potential benefits: the existing point of sale can serve as a presumably reliable fail over system; workers may require only minimal amount of training if any; secondary systems built on top of or also integrated with a POS can remain operational such as couponing systems, loyalty rewards programs, inventory systems, and the like; support for a wide variety of custom systems or setups; and other potential benefits.
The system and method may have particular applications to provide longitudinal tracking through a CV monitoring system across time and space and applying the generated modeled data in connection to a particular computing device encountered in the environment. In the case of assisting in the product checkout process, selection of items by a customer throughout the store can be tracked and then applied when the customer approaches a computing device such as a checkout station.

2. System

As shown in FIG. 1, a system for interfacing with a computing device from a computer vision system of a preferred embodiment can include a CV monitoring system integrated with an imaging system. The CV monitoring system is preferably used within a computing platform 100 managing a user interaction experience. The computing platform 100 will preferably include a data management system and other supplementary system elements of a computing platform 100, service, or application. The system may additionally include or interface with a user application and/or connected computing devices. In some variations, the system can include a proxy interface device 140.
A CV monitoring system 110 functions to process and generate conclusions from one or more sources of image data. The CV monitoring system no can provide: person detection; person identification; person tracking; object detection; object tracking; extraction of information from device interface sources; gesture, event, or interaction detection; and/or any suitable form of information collection using computer vision and optionally other processing techniques. The CV monitoring system no is preferably used to drive CV-based applications such as generation of a checkout list during shopping, tracking inventory state, tracking user interactions with objects, controlling devices in coordination with CV-derived observations, and/or other interactions.
The CV monitoring system 110 preferably operates in connection to an imaging system 112 installed in the environment. The imaging system 112 functions to collect image data within the environment. The imaging system 112 preferably includes a set of image capture devices. The imaging system 112 might collect some combination of visual, infrared, depth-based, lidar, radar, sonar, and/or other types of image data. The imaging system 112 is preferably positioned at a range of distinct vantage points. However, in one variation, the imaging system 112 may include only a single image capture device. The image data is preferably video but can alternatively be a set of periodic static images. In one implementation, the imaging system 112 may collect image data from existing surveillance or video systems. The image capture devices may be permanently situated in fixed locations. Alternatively, some or all may be moved, panned, zoomed, or carried throughout the facility in order to acquire more varied perspective views.
A computing platform 100 functions to coordinate or otherwise execute some application, service, or workflow. In a preferred implementation, the computing platform 100 facilitates executing a digital user interaction experience. For example, the computing platform 100 may facilitate a user interaction experience around or related to automated checkout and/or expedited checkout for customers. The computing platform 100 may include local computing resources at the environment to coordinate with devices present in the environment. The computing platform 100 may additionally or alternatively be remote computing resources wherein local computing systems in the environment communicate and coordinate over the internet (or other communication channel) with resources of the computing platform 100.
The computing platform 100 preferably includes or otherwise interfaces with the CV monitoring system 110, which functions to detect and model CV-people and/or other CV-based information. The computing platform 100 may additionally include or otherwise interface with one or more associative element monitoring system, which functions to establish an association with one or more associative element. The computing platform 100 can additionally include a data management system 130, which functions to manage and maintain user-records and/or user-accounts or other data modeling resources to support the objectives of the system.
The association monitoring system 120 functions as an associative or synchronization mechanism to establish one or more forms of association with a human and/or a computing device. In some variations, the association monitoring system 120 is part of another system element such as the CV monitoring system 110 and/or for an application service of the computing platform 100 to coordinate with a computing device instance (e.g., a user app). In other variations, the association monitoring system 120 can be an active sensing or interrogation system such as a wireless communication system, an RF/NFC interrogator, Bluetooth beacon, an audio sensor system, a specialized CV monitoring submodule, specialized application service, or other suitable type of system. In the variations that use an application instance to detect a device, the system may include an application service which functions to manage substantially real-time communication and interaction with application instances installed and operating on a user's device. The system may include one identification system, but may additionally include multiple types of association monitoring systems 120 that can be used redundantly or cooperatively.
A wireless communication system can function to monitor and detect the presence of device signatures in the environment. Multiple wireless communication units (e.g., WiFi routers, RF/NFC interrogators, and the like) can be positioned throughout the store. Their distributed location can be used to correlate the detection of a device signature with at least an approximate location. Bluetooth beacons could be used in a similar manner.
A specialized CV monitoring submodule can perform some form of CV-based analysis to assist in detecting. In one variation, this may include performing biometric identification or classification of CV-person. One specialized CV submodule could perform activity classification or modeling for a CV-person which could be matched to motion patterns detected from a special motion detection application service enabled on a device. In a similar manner, a CV monitoring submodule could detect particular gestures performed by a user. Another specialized CV monitoring submodule could detect visual patterns displayed by a computing device. A variety of types of CV monitoring submodules may be use.
An audio sensor system may function to detect audible signals in the environment. In one variation, audio signals detected by a computing device could be mapped to audio signals detected by microphones installed throughout the store. This could be used in determining approximate location. In some instances, specialized and potentially nearly inaudible signals could be output by the computing device and/or speakers in the environment. Detection of these unique signals by the device and/or microphones distributed in the environment could function to determine location and thereby selecting a candidate pool of CV-people for a computing device.
The data management system 130 functions to store user-records and/or user-account information. The system will generally include other elements to support the CV-based computing application and/or to provide a platform for CV-based computing applications implemented in multiple stores and/or for different use cases. The user-records and/or user-accounts may be linked or otherwise pre-associated with one or more types of identifying associative elements. For example, CV-based identification of an individual may generate an identity signature that is associated with a user-record. In another example, an identifier for a particular device instance may be associated with a user-record such that when that device instance is detected in the store, the mapped user-record can be identified and accessed.
User Applications and/or connected computing devices function as client devices that may participate in CV-based computing applications. In a preferred implementation, a user application of the store and/or operator of the CV monitoring system no can be made accessible to users of the CV monitoring system 110 (e.g., customers of a store). The user application can facilitate associating a user-account, providing biometric authentication through the device (e.g., fingerprint, voice recognition, facial identification or other forms of biometric authentication), collecting user input, and/or performing other tasks. A connected computing device can be computing devices that may be installed or provided within the environment. The connected computing devices may be customer service kiosks or stations, networked smart devices (e.g., smart lock, smart lights, etc.), a worker station, and/or any suitable computing devices.
A user application instance or a connected computing device are preferably addressable endpoints within the system such that the CV-system or an alternative component can transmit data or information can be communicated to the user application instance or the connected computing device. An SDK and/or API may be provided such that third party applications can establish such integration with a CV monitoring system 110 and/or platform.
The system may additionally include computing devices installed within the environment. In one variation, the environment-installed computing devices are configured and directly integrated with the system and computing platform such that they can be accessed and controlled at least in part. In another variation, the system may include a proxy interface device 140 that functions as a gateway or edge device that facilitates communication and interactions with outside devices. In this variation, existing computing devices like a POS system could be used with the system without making direct changes to the existing computing devices.
The proxy interface device 140 of a preferred embodiment functions to be a system controllable device/resource that interfaces with environmentally installed computing infrastructure. In some variations, the proxy interface device 140 impersonates an expected form of data entry. A preferred variation of the proxy interface device 140 is used to interface and augment an existing payment system. In one variation, the proxy interface device 140 is used expediting the checkout process by assisting in the entry of product information into an existing POS station. In another variation, the proxy interface device 140 may be used in assisting in delivering pre-authorized payment mechanisms so that a user could pay possibly without any explicit action (other than perhaps confirming payment/purchase).
Interfacing with an environment-installed computing device and/or a proxy interface device 140 would have applications in other areas as well. For example, in a restaurant, a digital ordering kiosk could automatically bring up a customized ordering interface (e.g., showing customer order history, starred items, personalized recommendations, loyalty reward options) based on CV-based identification of the user. In a gym, a user may be able to walk up to a device distributed through the gym and a personalized user interface can be automatically presented based on CV-based identification of the user and/or previously tracked activity in the gym. In an airport setting, a check-in kiosk could be controlled to automatically enter ticket information. These and other applications could be implemented through the system and method using a proxy interface device 140 and/or directly connecting to the appropriate computing device.
A proxy interface device 140 preferably emulates an input device for a target computing device. For example, the proxy interface device 140 could be a keyboard or barcode scanner emulator (i.e., product entry emulator). The proxy interface device 140 may alternatively communicate data that then uses an installed service to communicate with or otherwise interact with a target computing device.
A proxy interface device 140 preferably includes a communication module and a device interface module. The proxy interface device 140 may additionally be delivered as an attachment device encased within a body structure. The proxy interface device 140 may additionally include additional ports. In the checkout station use case, the proxy interface device 140 can include a product entry interface module so that the proxy interface device 140 can be connected between a barcode scanner and a checkout station.
The proxy interface device 140 communicatively couples the system (more specifically the CV-based monitoring system) with a computing device. A preferred implementation, the computing device is a point of sale checkout station, but the communicatively coupled computing device could be any suitable type of computing device such as an informational computing kiosk or computing devices for other purposes.
The communication module functions to allow the device to be remotely controlled. The communication module can preferably communicatively coupled with the CV-based monitoring system. The communication module is preferably a wireless communication module where the wireless communication module can include Wi-Fi communication module, a Bluetooth or RF communication module, a cellular communication module, or any suitable type of communication module. The communication module could additionally be a wired communication module wherein the proxy interface device 140 communicate over a wired or physical connection.
The communication module is preferably registered to the CV-based monitoring system. The communication module can additionally be registered in association with a target computing device. The location and/or target association could be registered during an enrollment period. As part of the enrollment process, the proxy interface device 140 could be registered and manually, automatically, or semi-automatically associated with that location and connected computing device (e.g., a POS station). In one variation, the proxy interface device 140 could include an active or passive identifier to announce itself to the CV monitoring system for registration and location determination.
The device interface module, which functions to communicatively connect or interface to one or more computing devices. In one variation this could be a wired connection. In one example, the proxy interface device 140 connects to a POS station through a port typically used by a barcode scanner. In some implementations it may replace a connection of a barcode scanner. In other implementations it may be connected to the POS in addition to one or more barcode scanners. In another variation, the proxy interface device 140 and/or the computing platform more generally can wirelessly communicate with a computing device like a checkout station.
The proxy interface device 140 may connect to a checkout station through the device interface module using a standard electrical interface like a USB interface or other data port. The proxy interface device 140 and more specifically the device interface module can generate and transmit the appropriate electrical signals for communicating with the checkout station. Product entry module in a preferred implementation can transfer the universal product codes (UPCs) of the modeled checkout list. For some POS stations this can be UPCs communicated as simulated keyboard entries. These can be transferred in some serial sequence such that from the POS station it appears as rapidly entered products. The product entry emulator may additionally include an interface to one or more product entry devices (like a barcode scanner or a keyboard) such that the product entry emulator can be used in line with such entry devices as shown in FIG. 13.
The product entry interface module can be a corresponding communication port to that used by the device interface module. This is preferably used so that use of the proxy interface device 140 preserves the option of the checkout station from using existing infrastructure like a barcode scanner. Accordingly, the product entry interface module could be a female USB port. Data received through this port may be automatically relayed through the device interface module to the connected computing device. Additionally or alternatively, data received through the product entry interface module can be captured and/or optionally processed. In a preferred implementation, UPC data collected through the product entry interface module can be relayed to the connected checkout station, but also communicated to the computing platform. This can be used as training data for CV, AI, and/or machine learning algorithms. Especially since, product information received through the product entry interface module will generally occur when there was an exception or issue with generating a checkout list.
The proxy interface device 140 can preferably be encased in a rigid structure. The proxy interface device 140 will additionally include a microprocessor or other form of processing unit along with memory, power system, and/or other suitable elements. The proxy interface devices 140 may be used for each checkout station. Alternatively, a single proxy interface device 140 may interface and manage multiple checkout stations.
The proxy interface device preferably includes user interface elements. The user interface elements are preferably provided to communicate with a user of the connected computing device. For example, the user interface elements can function to facilitate conveying system state and optionally collecting user input from a worker managing a POS station. User interface elements may additionally or alternatively be configured for use by other users like a customer.
In one preferred variation, the user interface elements include one or more forms of a user interface output. A user interface output could be a visual output, an audio output, a tactile output, and/or any suitable type of output.
A visual output could be some form of a visual indicators (e.g., LED lights or gauges) or even a graphical display. In one implementation, an indicator light(s) could signal when a checkout list is transferred/received for updating a POS system. For example, the display could blink green when a customer's selected items are automatically entered into a POS station thereby signaling to the worker that they do not need to scan items for entry and proceed to finalizing payment. Additionally or alternatively, the visual output could indicate when a checkout list cannot be automatically entered, if part of the checkout list can/was entered, and/or any other suitable exception. In another implementation, a graphical display could visually display the virtual checkout items that were relayed or suggested for being relayed to a POS station. The worker may even be requested to verify the list and select an option to enter them into the POS station. Changes could be made if a mistake was made. These changes could be relayed back to the CV-based monitoring system as feedback for quality control and monitoring enhancements.
An audio output could be used along with a visual output or by itself. An audio could be an audible tone or signal that triggers for different events. For example, there could be a tone or jingle that plays when a customer with valid checkout list approaches the “primary position” of the POS station (i.e., the person/group ready for checkout) which can indicate to the worker that they can skip the scanning/entry of items. Similarly, a tone may play when a checkout list is not available or not valid for entry, which can indicate to the worker that they should manually enter items selected by the customer.
The user interface elements could similarly include user input elements like buttons, dials, touch screens, microphone (for audio commands), and the like. These may be used in various ways to allow the worker to input information or communicate with the system. In one implementation, items are not automatically entered into a POS station until a worker positively confirms the items for entry.
A preferred variation of the proxy interface device is one that interfaces with a POS station for the purposes of relaying item information based on a checkout list generated by the CV-based monitoring system.

3. Method

As shown in FIG. 6, method for interfacing with a computing device from a computer vision system of a preferred embodiment can include detecting and tracking a human as a CV-person within an environment S110, establishing a user-record associated with the CV-person S120, modeling interactions of the CV-person as modeled data that is associated with the user record S132, detecting the CV-person in proximity to a computing device in the environment S140, and transmitting modeled data associated with the user-record to the computing device S150.
Block S110, which includes detecting and tracking a human as a CV-person within an environment, functions to establish a CV-person within an environment. Block S110 is preferably implemented through or in connection with a CV system, which could be part of a computing platform such as was described above. Detecting and tracking a human as a CV-person preferably includes collecting image data and applying person detection computer vision processing to the image data.
Image data is preferably collected and then processed by the CV system. When a human enters an environment with an operating CV system, the CV system can acquire the human as a CV-person. That CV-person can be tracked during the human's time in the environment. Continuous tracking may be maintained, but the system could additionally handle discontinuous tracking where tracking of a CV-person may be lost temporarily and then reacquired. Modeling of a CV-person is preferably abandoned, expired, or made inactive after the CV-person exits an environment.
Detection and tracking are preferably facilitated through various computer vision approaches. In one implementation, tracking may be achieved through a visual “lock”, wherein movement between frames of a detectable CV-person can be tracked. A physics or modeling system can be introduced to facilitate tracking the CV-person across frames and to recover when a user is temporarily or partially occluded.
Detection and tracking may additionally use human identification. Human identification can include biometric identification such as facial identification, gait identification, voice identification and the like. Human identification may additionally include more general characterizations of a human such as visual properties, clothing properties, height and weight estimations, age estimation, and/or possession properties (e.g., holding a bag, using an identifiable shopping cart, etc.). Human identification can be used to uniquely identify a human but may additionally be used to reduce the possible candidates of if not uniquely identifier the identities of humans within an environment. Tracking a CV-person could use human identification to assist in reacquisition of that CV-person.
In addition to detecting and tracking of a CV person, the CV-system can additionally provide additional CV-based analysis of the image data such as object detection, object-human interaction detection/classification, human gesture detection, and/or other suitable forms of CV-based modeling based in part on the image data. For example, in a shopping environment, the items selected by a CV-person and/or added to a cart of a CV-person can be added to a checkout list, which can be modeled and stored in association with that CV-person. The CV system can sense and generate predicative models based on image data and supplementary data sets (e.g., shopping history, user profile information, product information, etc.). The CV system may additionally work in combination with other sensing systems such as a smart shelving system with sensors, an RFID sensing system, 3D spatial sensing system, and/or any suitable type of environmental sensing system.
In an alternative variation, the method may operate based on detected and tracked mobile computing devices. The detected computing device will generally be a personal computing device such as a mobile phone, a wearable computer, or some other type of computing device. The detected computing device could alternatively be a non-personal computing device such as a computing device provided by the operators of the environment. For example, a smart shopping cart or basket may be used in place of a personal computing device. Detecting and tracking a mobile computing device may be used in combination or independently from detecting and tracking a CV-person. As such some variations described herein could alternatively be performed where a detected computing device (that will generally have some form of relationship to one or more users) can be used in place of a CV-person. For example, an application that enables user-managed scanning of items within a shopping environment may be detected and it's proximity to a checkout station tracked to facilitate transfer of a generated checkout list to the checkout station.
Block S120, which includes establishing a user-record associated with the CV-person, functions to link or set a user-record to be associated with a CV-person.
In one variation, the user-record could be set by default and used to collect or model CV-based observations of the CV-person. Exact data model architecture could be approached in a variety of ways and in some implementations, there may be no explicit distinction of a user-record from the CV-person. For example, the CV-person data model may serve as the user-record. Without additional association information, the identity of the corresponding human may not be identified or linked, personal computing devices of the corresponding human may not be linked, and/or other historical data beyond what was tracked in this session by the CV-based monitoring system may also not be linked.
In other variations, a user-record and/or a user-account may establish an association with the CV-person using a variety of associative/synchronization mechanisms. This may enable human identification (e.g., for the purposes of monitoring across visits to the environment), verified human identification (e.g., affirmative confirmation of the identity of a human), interfacing with another device instance (e.g., a computing device of the human), and/or access to a user-account. The variations described in illustrate optional association channels through which an interaction experience like checkout could be further enhanced or enriched.
Block S120, which includes establishing an association between the CV-person and at least one associative element, functions to synchronize and/or link a computer vision derived model of a human to at least one other data model related to the human. Preferably, at least one association is established to an associative element. This associative element may function as a human identifier or entity identifier. In some variations, this human or entity identifier could be token-based (e.g., like a device associated with the human), biological-based, or pattern-based (e.g., behavioral pattern detection). In general, the method is preferably executed so as to establish an association with a user-record and/or user-account.
Multiple types and/or instances of associations may be established for a particular human. In some variations, multiple associations may be established at different times, which can work to provide multiple factors of identification and/or to appropriately augment the confidence of a mapping between a CV-person and a user-record. These associations can additionally act as additional factors in having high confidence in other associations like when associating a CV-person with environment-installed computing devices like a checkout station. Some forms of associations may enable particular functionality. For example, restricted products like alcohol could be permitted for purchase by using a validated identity established through forming some form of human identity association.
Establishing an association with a user-record and/or user-account is preferably established through at least one identifying associative element like a device instance, application instance, and/or a human identifier. One or more associative mechanisms may be used to establish the association. With an association established to an identifying associative element, the identifying associative element can be used to indirectly establish an association with an appropriate user-record or user-account. The user-record or user-account can be stored and queried using a key, identifier, signature, or properties acquired from the identifying associative element. Accordingly, block S120 will generally include establishing an association between the computer vision modeled person and at least one associative element and thereby associating the computer vision modeled person and a user-record linked through the associative element. Effectively, the method establishes a three-element association between: a CV-person; one or both of a user-record and user-account; and at least one of a device instance, an application instance, and/or a human identifier.
An association to a user-record, which may specifically be a user-account, can expand the understanding of a CV-person to platform-managed data and/or to a historical record of the human entity. In an automated checkout implementation, payment information, shopping history, personalized loyalty program rewards, and/or other information may be exposed for use in combination with CV modeled data.
An association to a user-record is preferably made in connection with some identifying association such as with a device instance association or association with human identity.
An association to a device instance (or in some variations an application instance) can facilitate identifying a human associated device. The association with the device instance may further enable a communication link between the CV monitoring system and a device and/or application that is accessible to the corresponding human. An association to a human identity may enable authentication of the identity of the human corresponding to the CV-person. In one variation, an association to a human can be performed directly through CV-driven analysis of the CV-person. For example, biometric identification may be used. In another variation, authentication and more specifically biometric authentication may be facilitated by an accessible computing device such as a personal computing device or kiosk using biometric authentication.
Establishing an association with a CV-person can involve the use of one or more associative mechanisms for establishing associations. Different associative mechanisms may have different levels of reliability, security, and/or be usable for adding associations to different associative elements. A subset of these associative mechanisms may be classified as system initiated associative mechanisms and another subset of associative mechanisms may be classified as user-initiated associative mechanisms. Establishing an association may involve executing variations of such associative mechanisms as shown in FIG. 7.
System-initiated associative mechanisms can leverage natural conditions without depending on some form of active user participation to work. System-initiated associative mechanisms can be used in automatically establishing at least part of an association. They will generally operate in the background and may be transparent to a user. System-initiated associative mechanisms may be used in establishing an association automatically—possibly without active participation by a human at the time of association. Three potential types of system-initiated associative mechanisms can be biometric, passive, and active.
A biometric type of system-initiated associative mechanism preferably uses biometric input collected in real-time from a CV-person in the environment. This biometric identification can preferably be performed remotely through the image data used by the CV monitoring system. Biometric identification may be used in combination with other associative mechanisms to increase the confidence in biometric identification. For example, using device signature detection and/or gross device location information, the candidate pool of user identities can be reduced such that the confidence in biometric identification can be made more resilient. A facial recognition CV process could be configured to identify a face out of a pool of 300 candidates near the store location as opposed to millions of potential candidates encountered at some time across many sites.
A passive type of system-initiated associative mechanism can be an “always on” technology that is typically exposed by a computing device carried by a human. These signals may be unique to each device or at least provide distinguishing features within a limited sample of devices, and therefore can be used to form connections between devices and CV-persons with a relatively high degree of certainty. Passive associative mechanisms may not establish a positively confirmed association in some situations since devices can be shared or stolen, but passive background device detection may be used to establish a probabilistically likely association and/or to narrow the scope of associative elements for other association techniques like the automatic biometric analysis above. Such signals can include WiFi, Bluetooth, MAC addresses, cellular signature and the like. Such signals may not be 100% accurate associations because smart devices can be shared or stolen by different humans. However, such signals can at least be useful with humans during a single visit. They may additionally be useful device signatures for humans with a personal computing device but without an installed application and/or without a user-account. For example, a human without installing a user application on their smart device can have a smart device passively facilitate associating a CV-person to a device instance and by association a user-record with purchase history and payment information.
An active type of system-initiated associative mechanism may be some user installed or initiated technology, which, in some cases, can affirmatively link CV-people with a computing device. This variation of system-initiated associative mechanism may operate without explicit device usage by a human while in the store. However, they may depend on installing an application, installing or enabling a service. Bluetooth LE beaconing, background app refresh, geofencing, GPS or relative GPS, other location services, and/or other device functionality may expose signals by which a device could be detected and associated with a user-record or user-account.
Use-initiated associative mechanisms can include an associative mechanism dependent on user action. One variation can be device-based initiation. Other forms of user-initiated associative mechanisms can be various forms of user check-ins.
Some preferred associative mechanisms could include detecting a human associated device instance S121 and/or collecting real-time image data of a CV-person in the environment and applying a form of biometric identification S125, and/or receiving registration at a registration device S126. Some preferred variations of detecting a human associated device instance S121 can include: detecting a device signature exposed by devices carried by a human S122, detecting a device usage-mechanism explicitly or implicitly enabled by a user S123, and establishing an association during a pairing process of a device instance S124.
Block S121 which includes detecting a human associated device instance functions to use the presence of an identifiable device to establish an association. Block S121 will generally involve: through a device identification system, detecting a device instance coincident to the CV-person and then selecting a user-record by an identifier of the device instance. This functions to associate the CV-person, the device instance, and the user-record. Here, the device instance could be some human identifiable device such personal computing device like a smart phone, a personal computer, a wearable computer, and the like. The device instance could alternatively be an instance of an application installed on a device, a browser session, or any suitable client detectable and associated with a user. In some variations, the device instance could be a shared environment-installed computing device. A shared computing device will have shared usage by multiple users at different instances. A shared computing device may be directly used by the human when establishing the association (e.g., an informational kiosk or self-checkout station). The shared computing device may alternatively be used to serve a particular human when establishing the association (e.g., a worker-staffed checkout station).
Here, the condition of coincidence characterizes some form of associated proximity, which could include spatial proximity and/or temporal proximity. A device instance may be spatially coincident when the device instance is detected in a location that corresponds to a modeled location of the CV-person. Detection of a device is preferably isolated to a particular region or location. This location can be mapped to a corresponding location in the environmental model of the CV-system. If a CV-person is detected in the corresponding location, an association can be established between the device and the CV-person. Depending on the spatial resolution of either the device instance or the CV-person, different thresholds of proximity may be used. In some cases, gross spatial proximity (e.g., detected within 50 feet range of each other) can be sufficient such as when the pool of possible candidate pairings of CV-people and device instances is not high. A device may be temporally coincident when the device instance is detected within a similar time window. For example, if a device instance is detected in the environment within one minute of detection of a CV-person then those could be paired.
In a crowded environment, device detection may occur at multiple locations in an environment and a CV-person association to a device instance can be established through longitudinal analysis of past instances of device detection. This can include looking at the tracked movements of a device instance, analyzing the sets of candidate device instances and CV-people, and/or using other suitable techniques. For example, a device may be detected at the entrance of a store. However, there may be multiple candidate CV-people so a confirmed association may not be established initially. The same device instance may be detected at a second location. Even if the number of candidate CV-people is still large, the CV-people detected at the entrance and the second location can narrow the candidate CV-people.
Detecting a device instance may be used in identifying and selecting a corresponding user-record associated with the device instance. In one variation, the device instance may provide some distinguishing key or identifier by which a user-record stored in association with that identifier can be found and accessed. In this variation, the detection of a device instance can be used to re-establish a previous user-record. This accessed user-record can be merged with or replace a current associated user-record.
Furthermore, a device instance may be used in establishing an association with a user-account. In some cases, the application instance may be associated with a user-account (e.g., if a human has signed into their account within the application), wherein, establishing association to a device instance will by association establish a user-account association. Accordingly, using a device instance to establish an association with a user account may include authenticating a user-account at the device instance; selecting a user-record based on the user-account authenticated on the device instance when the device instance is detected. The computing platform will generally have access to the device instance and therefore could map the user-account logged into the particular device instance with the CV-person.
Similarly, a device instance and/or an application instance can facilitate human identification and so may be establish a validated association with a human. This human association with the application instance can be transferred to an association with a CV-person as well. For example, fingerprint identification, facial identification, voice identification, or other forms of biometric identification performed within the application instance can establish human identity association.
Detection of a device instance may be performed through a variety of approaches. As discussed above, the block S121 may include detecting a device signature exposed by devices carried by a human S122, detecting a device-usage mechanism explicitly or implicitly enabled by a user S123, and/or establishing an association during a pairing process of a device instance S124.
Block S122, which includes detecting device signatures exposed by devices carried by a human, functions to facilitate passive background device detection. Passive background device detection is preferably used in establishing an association to a device instance through exposed communication device signatures. Device signatures preferably include communication signals from a device that are generally unique to a device and therefore can be used to form associations between a device instance and a CV-person. A device signature can be detectable properties or identifiers exposed over Wi-Fi, over Bluetooth, through MAC addresses, through cellular signals, and/or other exposed emissions. Block S122 can preferably operate as a system-initiated associative mechanism, and may be used to assist providing enhancement functionality to users even if they are not registered or enrolled with a user-account. As a user-initiated association, the user may set some option to enable (or disable) the user of device signatures. If a device instance was previously associated with a device-record or more preferably a user-account, then the device signature can be used in establishing associations to other associative elements. Block S123, which includes detecting a device usage mechanism explicitly or implicitly enabled by a user, functions to facilitate user-enabled background device detection. There are various options that when activated by a user will facilitate automatic establishment of associations of that particular user. In one variation, a user may explicitly enable geofencing to automatically detect when they enter a store. In another variation, a user may implicitly enable a detection mechanism by registering a user-account to access Wi-Fi in the environment. In another variation, a user may enable kinematic monitoring for synchronizing human motion to CV-person motion. These options may depend on a user installing an application and/or enabling or confirming particular device permissions.
User-enabled background device detection can be performed substantially similar to passive background device detection described above. However, user-enabled background device detection may use detection approaches like Bluetooth LE beaconing, background app refresh, geofencing, LED positioning, GPS, relative positioning in the environment, and the like.
With techniques like Bluetooth beaconing and geofencing, the system may be actively notified when the device instance is present in the environment. With knowledge of the presence of a device instance, that device instance and previously associated humans, user-accounts, and/or user-records may become candidates for associations with a CV-person.
When using Bluetooth beaconing, detecting the device instance can include detecting location and/or presence of an identifiable device instance. When Bluetooth beaconing is used in multiple locations, a CV-person can be mapped to and associated with a device instance when there is a unique correspondence of coincidence across a set of detection events as shown in FIG. 8. When there is only one CV-person in the region of a first beacon during a first detection event, then the detected device instance and CV-person can be associated. When there are multiple people during multiple detection events, a CV-person can be associated with the device instance if the CV-person is the uniquely present CV-person in the regions of the various beacons at the time of the detection events.
When using location detection, detecting the device instance can include detecting location of a device instance within the environment, and associating the device instance to a CV-person modeled with a corresponding location as shown in FIG. 9. Multiple samples of location and/or the change in location can additionally be used. In a manner similar to the Bluetooth beaconing approach, sampling across many regions can be used to reduce a pool of candidate CV-people down to one selected CV-person.
In a kinematic monitoring variation, the method may include, at the device instance, sensing device motion and/or orientation. This device motion and/or orientation can then be used in a variation where block S123 includes detecting an associated device instance with device motion that satisfies a synchronization condition with activity of the CV-person. In other words, motion of the device instance is matched to CV-person motion. An inertial measurement unit which can include accelerometers, gyroscopes, magnetometers (for direction), and/or other sensors may detect motion like walking motion, stops and stops, hand motion, body orientation changes, and the like. By comparing these actions to motion and orientation properties of a CV-person, a CV-person can be linked to a corresponding device instance. Use of motion as a synchronization mechanism may be dependent primarily on coincidence in time and may not factor in location though location can additionally be used.
In one variation, establishing an association may initiate confirming of the association, which functions to trigger user affirmation to establish an association. Confirming an association can include requesting confirmation through communication to a human (generally through a device instance). In one variation, an association may include a confirmed or possible association to a device instance. That device instance may have a communication address (e.g., phone number, application instance identifier, push notification endpoint, etc.). A message or notification can be transmitted to the communication address, and a human can confirm or deny the request. Upon receiving affirmative confirmation, the method can proceed with establishing the association. In another variation, the possible association may involve a user-record or a user-account, which may have a configured communication address like a phone number of application instance identifier. A message or notification can be similarly transmitted.
Confirmation may request explicit confirmation by a human. For example, a human may receive a text message saying, “Looks like you are at the supermarket. Replay 1 for yes and 2 for no”. A confirmation may alternatively notify a human but require explicit denial to cancel/prevent the association. For example, a human may receive a push notification saying “Enjoy your shopping experience today! Select cancel if you received this in error”. A confirmation may alternatively be automatically approved if a human selects a user-account option to not confirm new sessions. Confirmation could similarly be applied before performing particular interactions that depend on an established association. For example, a user may receive a push notification on an application asking if they want to automatically push checkout list information to a checkout station when they are within proximity.
Block S124, which includes establishing an association during a pairing process of a device instance, functions to have a device or more preferably an application facilitate establishment of an association. In some variations, a physical computing device may be designed to enter a pairing mode or continuously perform some pairing action. The pairing mode may be configured to generate an identifying signal through a user interface output of the device instance. As will be described below, the graphical display and speakers may both generate signals usable to detect a device. Similarly, sensed data may be collected and used during the pairing mode. In other variations, an application instance installed on a user device can be actively involved in performing actions that may enable unique detection of that application instance by the CV-system and thereby be associated with a CV-person. Herein, the pairing process of an application instance is described but it similarly applies to a device instance.
The pairing process is preferably initialized when the application instance becomes active (e.g., is brought into the foreground of a modal operating system). Activation of the application instance can result in the application instance registering the pairing state of the application instance with the CV monitoring system such that the CV monitoring system can begin to cooperatively establish an association. Once in the pairing mode, one or more signaling mechanisms may be used. The examples of signaling mechanisms can include use of a camera, a display or visual output, an audio system, an inertial measurement unit (IMU), application interactions, and/or stored data.
As shown in FIG. 10A, an application instance can use the camera of the computing device to capture an image that can then be used to determine position. In one variation, various visual identifiers (e.g., a QR code or data encoded image) can be positioned throughout the store. Capturing image data from the application can be used to determine the position of the human operating the application when the image data contains the visual identifier. In a related variation, the human operator of the application may be directed to take a picture of a cart of selected items or some other artifact, which can be used as an identifier.
In another variation, the camera may be activated in the background with the goal of observing the scene. This activation of the camera may be based on detected orientation of the device (e.g., is the device being held up by human). Observations of the scene can act as a visual signature that can be mapped to images captured by the CV monitoring system as another approach to locating the device within the environment. For example, an application instance may capture images while the user is using the application. If an image of a shelf of items is captured, that image can be used to determine the location of the user within the store and thereby associate the application instance with a CV-person. Visual identifiers may be integrated into the environment such as on the ground, ceiling, shelving, walls, and the like to enhance mapping of visual signatures and positioning within the environment.
As shown in FIG. 10B, an application instance can use the display or other mechanisms for visual output in pairing an application instance with a CV-person. In a preferred variation, the display of a smart device can be set to display a visual identifier. The visual identifier can be a static graphic but could alternatively be a time variable visual display. When displaying the visual identifier, the application instance communicates with the system so that detection of that visual identifier by the CV-system can be associated with the application instance. The imaging system used by the CV monitoring system will preferably be able to identify the displayed identifier and map an appropriate CV-person to the application instance. A visual identifier could alternatively be produced by other device elements such as an LED light.
As shown in FIG. 10C, an application instance can use the audio output in pairing an application instance with a CV-person. An audio signal is generated with a unique identifier. Detection of that audio signal can then be used to determine location and then used in selecting candidate CV-people. The audio signal is preferably in a frequency range on the edge or beyond the hearing range of a human, but could also be a sound audible by a human. In one variation, the application instance generates the audio signal and microphones distributed within the environment can detect the audio signal. The microphones are associated with a general region within the environment or may be used in triangulating a more precise location. In another variation, speakers distributed across the environment generate audio signals that can be detected by an application instance. Each speaker preferably produces a unique audio signal such that the detected audio signal can be mapped to a positioned speaker and/or a location. Detection of multiple audio signals may be used to triangulate a more precise location.
As shown in FIG. 10D, an application instance can use movement of the device as detected by an inertial measurement unit in pairing an application instance with a CV-person. The inertial measurement unit can include a digital accelerometer, gyroscope, magnetometer, and the like. During a pairing mode, the movements, orientation, and/or direction can be monitored to establish human movements. The CV monitoring system can similarly detect CV-person movements for candidate CV-people. CV-people with corresponding movement patterns at a similar time can be associated with the device instance.
As shown in FIG. 10E, an application instance can use application interaction as a mechanism for pairing an application instance with a CV-person. An application interaction may be used by detecting human operation of the application and using the CV monitoring system to detect candidate CV-people performing similar actions. Application interaction may be used passively, which preferably use normal interactions with the application instance as CV-detectable actions. For example, the application instance may require a human to perform a check-in action to use an automatic self-checkout feature. The act of pulling out a smart device and pressing a check-in button on the application instance can be detected as actions performed by a CV-person. Additionally, the timing of activating the check-in button as detected by the application instance can be used to determine CV-people performing that action in synchronization with that act.
During a pairing mode of the application instance, block S120 can include directing a physical action by the user and detecting a corresponding physical action of the CV-person. In matching the sensed motion at the device instance to modeled motion of the CV-person, the device instance can be associated with the CV-person. For example, Application interactions may be actively directed by the application instance where an application instance instructs a human to perform some action or gesture such as waving his or her hand, holding out the smart device, pointing at an object, or any suitable action. These human gestures may be more detectable than interactions with the application instance. The timing of the actions can be orchestrated by the application instance. Additionally, the nature of action may be controlled and changed to help in establishing the association. For example, different types of actions may be directed so that the type as well as location and timing can better establish an association with high confidence. Device movement as discussed above can similarly be used in combination with directed actions.
Establishing an association during a pairing process of a device instance may additionally or alternatively use collected information from the application instance. When a pairing mode is activated, preferably by the human, then the system may begin using stored information associated with the application instance and/or account to establish an association. As one specific example, if a customer has entered a shopping list into an application, then the CV monitoring system can compare that shopping list to the checkout lists of current CV-people. A CV-person detected to have some subset of the shopping list may be associated with that application instance as shown in FIG. 10F. Shopping history, favorited products, and/or other information can similarly be used.
Block S125, which includes collecting real-time image data of a CV-person in the environment and applying a form of biometric identification, functions to facilitate automatic biometric analysis of a CV-person. A variety of types of biometric identification techniques may be used such as facial identification, gait identification, retina identification, physical property identification (e.g., height, weight, gender, etc.), and/or other forms of biometric identification. Biometric analysis may be used to establish a human association when a unique human identity is determined. Automatic biometric analysis may additionally establish associations with other associative elements like a user-record.
As an alternative or additional approach to biometric identification, biometric identification sensing capabilities of a computing device may be used when a computing device can be communicatively coupled and associated with a CV-person (e.g., using one of the approaches above). This variation can include, at the device instance, receiving biometric identification; and associating human identity confirmation with the modeled person and user-record. Receiving biometric identification can act to confirm or specify the human identify of the current user of the computing device. Receiving biometric identification could include checking a fingerprint, facial pattern, voice, vital sign (e.g., heart rate, respiration rate), and/or any suitable biometric identification. These may be provided on a personal computing device or some shared computing device like a check-in kiosk in the environment.
Block S126, which includes receiving registration at a registration device, functions to establish an association during active registration of a user-record, user-account, human, and/or application instance. The registration device can be a kiosk or check-in station installed in the environment. The human will preferably perform some interaction with the registration device to establish an association. The mode of registering at the registration device can take many forms such as swiping/scanning a card (e.g., a payment card), presenting NFC device, QR code or barcode scanning, biometric scanning, entering of account credentials, entering a pin code, and/or any suitable action. A registration device could additionally include biometric sensors for establishing human associations. Some of these registration techniques may not require possession of a device instance and/or other physical tokens. This may provide an option for humans that want to participate in the system without using a physical device. Location of the registration device(s) and proximity of a CV-person during the time of registration can be used to associate with a CV-person. In one alternative, the method may use registration regions where some registration can occur without a physical registration device.
The mechanism for establishing an association can be one or more of the mechanisms described herein. In some cases, multiple association mechanisms are used in combination to narrow a pool of candidate CV-people to a single confirmed CV-person. The method can additionally redundantly confirm associations as a form of multi-factor authentication to enhance accuracy, reliability, and/or security. In one variation, establishing an association can include progressively attempting a sequence of association mechanisms. When one association mechanism does not succeed, a backup association mechanism can be used. The sequence of association mechanisms may start off with less intrusive user experiences and then escalates to association mechanisms with higher reliability but possibly more reliance on human participation. In one example shown in FIG. 11, establishing an association to an application instance can initialize by passively using the location of the application instance and identification of a displayed static graphic. If an association is not established, then the application may escalate to displaying an animated graphic with instructions to hold their phone outwards with display up. If an association is not established again the application instance can start an audible jingle while flashing an LED light. If an association is still not established, the application instance can direct the user to use a check-in station. From the user perspective, this pairing process can be presented as a seamless pairing process that slowly builds up. Ideally, an association is established quickly so that most users do not experience the more obvious association mechanisms.
There may also be different modes of associations that may depend on the associative elements, mechanism for establishing associations, and/or user selected options. There could be an anonymous association where a CV-person is treated as a generic user. There could be an incognito association mode, where information and capabilities enabled by associations to a user-record, user-account, application record, and/or a human may be used but where the session is not committed to the data records of those associations within the system. There could also be various forms of confirmed identity associations. For example, an association can be an association with confirmed human-identity wherein secure permissions can be enabled such as those that traditionally require review of form of physical identification.
Associations are preferably established prior to performing some action requiring permission confirmed through the association. In a preferred implementation, associations are established in response to nearing or entering an environment. However, the method can support retroactively applying an association. As an illustration of the flexibility in establishing an association, the method can support establishing an association immediately prior to the human entering the store and being acquired as a CV-person, immediately before a transaction during an automatic self-checkout, and/or in conjunction with the transaction. Associations can additionally be built up during the duration of the time that the human is in the environment, wherein establishing an association can include wherein establishing an association through multiple associative mechanisms performed at different locations and times in the environment. For example, a CV-person may initially be associated with a user record, then an application instance can be added and then a human association. Associations can additionally happen after a CV-person has left the store where actions like a billing transaction can happen in absentia. This can even happen over multiple visits to the environment by a human. For example, a shoplifter that later creates a user-account may have previous activities and illicit behavior retroactively associated with the new user-account.
Block S132, which includes modeling interactions of the CV-person as modeled data that is associated with the user record, functions to sense information about the CV-person. In one preferred variation, modeling interactions of the CV-person comprises modeling a checkout list of items selected by the user in the environment. Modeling a checkout list, functions to generate list of items selected by a human. The checkout list is at least partially based on CV-based monitoring of the CV-person. The CV-based monitoring can be stored in association with the user-record such that established associations with a computing device can leverage user-record information such as current checkout list/virtual cart. As discussed above the CV monitoring system can additionally provide additional CV-based analysis of the image data such as object detection, object-human interaction detection/classification, human gesture detection, and/or other suitable forms of CV-based modeling based in part on the image data. In a shopping environment, the items selected by a CV-person and/or added to a cart of a CV-person can be added to a checkout list, which can be modeled and stored in association with that CV-person. The CV monitoring system can sense and generate predicative models based on image data and supplementary data sets (e.g., shopping history, user profile information, product information, etc.). The CV monitoring system may additionally work in combination with other sensing systems such as a smart shelving system with sensors, an RFID sensing system, 3D spatial sensing system, and/or any suitable type of environmental sensing system. Checkout lists may alternatively be generated and associated with a user-record in other ways. For example, a customer may use an application to scan items for purchase and self-populate a checkout list. In this variation, the CV-person may not be tracked throughout the environment and only in the region of a computing device of interest.
Block S132 is preferably part of managing state and/or modeled data in association with the user-record S130, wherein the origin or nature of the modeled data can vary. Accordingly, the method may additionally or alternatively include managing user-record data in association with the user record S134, collecting user input through an associated application instance of the user-record S136, and/or other suitable variations.
Block S134, which includes managing user-record data in association with the user record, functions to use other types of data associated with a user-record. The data may originate from user history data, user preference/configuration data, data or state information from a device associated with the user-record/user-account. In one example, the user-record data could include payment mechanism data which can be transferred or otherwise used with a payment-oriented computing device so that a person could pay by approaching the computing device. In one preferred implementation, the user approaches the payment device, the balance is entered in some manner (possibly also provided through the CV monitoring system), the user is presented with the checkout information and asked to confirm payment using this synchronized payment mechanism, and a payment transaction is executed with the payment mechanism if the user provides confirmation.
Block S136, which includes collecting user input through an associated application instance of the user-record, functions to use user interface input, sensed data, or other data generated at a user-controlled device to be delivered to a target device. In this variation state and/or data from a user device is relayed to a target device. For example, a user may walk up to a payment kiosk, and the payment kiosk may ask for some form of user input which is then requested and provided through an application operable on the user's personal computing device.
A similar variation of the method can be applied so that user interface input, sensed data, state information, or other data generated at a target computing device can be delivered to a target device. In this variation state and/or data from a target device is relayed to a user device/application. Some variations may support a bi-directional flow of data between two computing devices (e.g., a target device and a user device).
Block S140, which includes detecting the CV-person in proximity to a computing device in the environment, functions to establish an association between a detected computing device and a CV-person. This preferably establishes a multi-element association of a detected computing device, a CV-person (and CV-modeled data or events), and a corresponding user-record. This may further be used to establish an association with another computing device (e.g., a personal computing device of a user), a user-account, and/or a human-identity. These different types of associations that can be dynamically established can offer various forms and interaction capabilities.
The computing device can be a kiosk, a computing station, networked smart device (e.g., smart lock, smart lights, smart scales, etc.), and/or any suitable computing device. As discussed, in a preferred implementation, the computing device in the environment can a checkout station used to perform a checkout process. At or near a checkout station there is generally a line of people waiting to checkout. Various rules, heuristics, and autonomous processes can be used in checking a proximity condition. For a checkout station there will generally be a single CV-person that can be associated with at one time. This will generally be the CV-person modeled within some defined payment region and/or closest to a defined payment point of the checkout station. The locations of the checkout stations may be preconfigured within the CV monitoring system so that analysis on CV-people in near proximity can be isolated to that segment of image data. Alternatively, a computing device may be dynamically detected and located within the environment. Furthermore, the identity of the computing device is detected or known so that association can be established with the correct computing device.
In one implementation, a defined space, possibly marked on the floor could be used in indicating who is able to associate and synchronize with a checkout station.
As another heuristic, the conveyor belt of a checkout station, the items on that checkout station, and the divider used to mark separation between customers could be monitored, analyzed, and used in determining who has the primary position of proximity to the checkout station. Preferably, as the items are moved off the conveyor belt, the next customers items move towards the front of the conveyor belt. Detecting removal of the divider for example, could be used trigger associating that corresponding user-record and synchronizing the checkout list with the checkout station. As one variation, a worker may provide some form of explicit user input to mark when and who should be processed and synchronized with the checkout station.
As another variation, association with a checkout station may not be limited to a single user-record/checkout list. Multiple individuals in a defined region may have a checkout list synchronized with a checkout station. For example, a family shopping together would be represented as multiple CV-people, but those modeled interactions and resulting checkout lists may all synchronized with a checkout station when the family moves to a defined payment region.
Block S150, which includes transmitting modeled data associated with the user-record to the computing device, functions to communicate with an associated computing device. This synchronization of data preferably synchronizes data specific to the CV-person and/or device instance. Data and information can flow in one direction from user-record related data, but data and information could similarly flow in multiple directions. As the CV monitoring system will preferably be used simultaneously by multiple distinct humans, this synchronization with a computing device is preferably simultaneously orchestrated by a computing platform across multiple humans.
In one variation, the computing platform can communicate the modeled data directly to the computing device through a wireless or wired connection as shown in FIG. 12. In this variation, the computing device may have an exposed interface protocol through which the computing platform can use.
In another variation, the computing platform can communicate the modeled data indirectly to the computing device by communicating modeled data to a proxy interface device as shown in FIG. 13. The proxy interface device preferably relays that data to the targeted computing device. In some variations, the proxy interface device can use an exposed interface to gain interactive access to the computing device.
In a preferred implementation, the computing device can be a proxy interface device and the modeled data can include a checkout list. Upon receiving the checkout list from the computing platform, the method will preferably include relaying an itemized checkout list data to the checkout station. The proxy interface device can simulate a barcode reader and enter UPC information. The checkout list is preferably received at the proxy interface device, and then it may be relayed through a device interface module as a series of UPC codes entered as simulated keyboard entries. This functions to emulate a barcode scanner or a keyboard for product code entries. Other suitable protocols could alternatively be used.
As discussed above, the proxy interface device may additionally connect to a product entry device like a barcode scanner or a manual product code entry tool like a keyboard. The proxy interface device can be communicatively coupled with the product entry device. Upon receiving item data from the product entry device, the method can include relaying the item data to the checkout station. This functions to maintain normal operation of the product entry device—the scanning information passes through. Additionally, the proxy interface device can process and use the item data received from a product entry device. Accordingly, the method may additionally include upon receiving item data from the product entry device communicating the item data to the computing platform.
In another variation, the modeled data could be payment details linked to a user-record (or more generally a user-account). The payment details could be transferred as shown in FIG. 14. In some variations, the computing device and/or the proxy interface device can be a digital payment transmitter which is configured to transfer payment details (e.g., payment mechanism tokens) digitally at a point-of-sale device. In another variation, as with the barcode scanner emulation described above, a proxy interface device may emulate some payment mechanism behavior such as emulating an NFC payment mechanism or emulating a present credit card.
The proxy interface device is preferably addressable based on the desired checkout station. In the event a proxy interface device connects to multiple checkout stations, communication with the proxy interface device can specify a checkout station identifier/address.
In connection to the proxy interface device, the method may additionally include altering the proxy interface device depending on the state of association. Updating a user interface output at the proxy interface device can facilitate communication of machine state to a human (e.g., worker and/or a customer). In general, the user interface output can be used to indicate if and when modeled data was transmitted, the nature of that transferred data, and/or other information. In one implementation, a user interface output can communicate if modeled data (e.g., a checkout list) was successfully transferred and/or if modeled data was not successfully transferred. For example, a checkout list is preferably successfully transferred when the CV-based modeling of the checkout list has a high confidence level. A checkout list may encounter an exception or error if modeling of the checkout list has low confidence for one or more items, if one or more products is ineligible for automatic entry, if the user is not permitted to or does not have a checkout list, and/or any suitable issue.
In particular, the method can include in response to receiving a checkout list at the proxy interface device triggering a user interface output. A user interface output is preferably triggered in response to receiving a checkout list at the proxy interface device. The user interface output could be an audible tone, activation of some display state, graphical representation of the modeled data on a graphical display, or other suitable forms of output.
Triggering user interface output can include, If the transmitted checkout list qualifies, outputting a confirmation signal through the user interface output, which functions to communicate a successful checkout list transfer. In some implementations, the output may be generated when a checkout list is received at a proxy interface device. Receiving worker and/or customer confirmation may be requested before finalizing transfer to the checkout station for processing. In other implementations, the user interface output may be triggered when the checkout list is successfully transmitted or delivered to the checkout station.
Additionally, triggering the user interface output comprises of, if the checkout list transfer has an exception, outputting an exception signal through the user interface. This functions to signal to a user that there was an issue. In some cases, the issue can be resolved, and this may be presented through the user interface output. For example, an exception may indicate that certain classes of items need to be manually entered. Items such as restricted items like alcohol, difficult to identify products like produce, items sold by weight, and other types of items may trigger exceptions. In other instances, a checkout list may not be usable or available for use and a signal could be triggered indicating this. When not able to resolve the exception, the worker may use that signal to proceed with a manual product entry checkout process.
The method additionally can additionally include disassociating a CV-person and a computing device. A CV-person is preferably disassociated when the CV-person loses priority status. In one variation, the priority status may be lost when the human associated with the CV-person leaves a particular region. In another variation, the priority status may be lost when a second human is assigned priority status if for example, if the second human moves to be the closest human. Various heuristics, rules, and detection processes may be used in determining which individual or group of users should be treated as the primary. In some variations, though multiple users may be simultaneously synched with a target computing device. For example, if a group of four people all select items in the store, those individually detected checkout lists can be grouped for checkout by the associated humans all entering a checkout region next to a checkout station.
As an exemplary application of the method, a first user will enter the store and select various items as the user walks through the store. The CV monitoring system (or some other sensing system) can track a checkout list of the first user that represents predicted “cart” contents for the user. Here the cart contents may be stored in the cart but could alternatively be store in a basket, held in bag, held in the hand, stuck into pocket. When the first user is done, they can approach a checkout station. Other users may be in front of the first user, and those users may have similar process applied to their checkout experience. In some cases, these other users may have a tracked checkout list automatically entered into the checkout station for expedited checkout, part of the checkout list automatically entered, and/or assisted in any suitable manner.
The systems and methods of the embodiments can be embodied and/or implemented at least in part as a machine configured to receive a computer-readable medium storing computer-readable instructions. The instructions can be executed by computer-executable components integrated with the application, applet, host, server, network, website, communication service, communication interface, hardware/firmware/software elements of a user computer or mobile device, wristband, smartphone, or any suitable combination thereof. Other systems and methods of the embodiment can be embodied and/or implemented at least in part as a machine configured to receive a computer-readable medium storing computer-readable instructions. The instructions can be executed by computer-executable components integrated with apparatuses and networks of the type described above. The computer-readable medium can be stored on any suitable computer readable media such as RAMs, ROMs, flash memory, EEPROMs, optical devices (CD or DVD), hard drives, floppy drives, or any suitable device. The computer-executable component can be a processor, but any suitable dedicated hardware device can (alternatively or additionally) execute the instructions.
As a person skilled in the art will recognize from the previous detailed description and from the figures and claims, modifications and changes can be made to the embodiments of the invention without departing from the scope of this invention as defined in the following claims.

Claims

We claim:

1. A method comprising:

through a CV monitoring system, detecting a computer vision modeled person and setting a user-record in association with a modeled person, the CV monitoring system being part of a computing platform;

modeling interactions of the modeled person as modeled data associated with the user-record;

detecting the modeled person in proximity to a computing device in the environment and thereby establishing an association of the computing device with the modeled person; and

after establishing an association of the computing device with the modeled person, transmitting at least a part of the modeled data from the computing platform to the computing device.

2. The method of claim 1, wherein modeling interactions of the modeled person in association with the user-record comprises of modeling a checkout list of items selected by the user in the environment; and wherein the modeled data is at least part of the checkout list.

3. The method of claim 2, wherein the computing device is a checkout station device.

4. The method of claim 2, wherein the computing device is a proxy interface device that is communicatively coupled to a checkout station; and further comprising upon receiving the checkout list from the computing platform, relaying an itemized checkout list data to the checkout station.

5. The method of claim 4, wherein the proxy interface device is communicatively coupled with a product entry device; and wherein upon receiving item data from the product entry device, relaying the item data to the checkout station.

6. The method of claim 5, further comprising, at the proxy interface device, upon receiving item data from the product entry device, communicating the item data to the computing platform.

7. The method of claim 4, wherein the proxy interface device is communicatively coupled to the checkout station with a wired connection.

8. The method of claim 4, wherein in response to receiving a checkout list at the proxy interface device triggering a user interface output.

9. The method of claim 8, wherein, triggering the user interface output comprises of, if the checkout list qualifies, outputting a confirmation signal through the user interface output.

10. The method of claim 9, wherein triggering the user interface output comprises of, if the checkout list transfer has an exception, outputting an exception signal through the user interface.

11. The method of claim 2, wherein the modeled data is wirelessly communicated to the proxy interface device.

12. The method of claim 2 further comprising storing a payment mechanism in association with the user-record, wherein the transmitted modeled data is the payment mechanism.

13. The method of claim 12, wherein the computing device is a digital payment transmitter.

14. A system comprising:

a computer vision monitoring platform configured to:

detect a computer vision modeled person associated with a user-record,

generate a checkout list for the user-record,

detect the modeled person in proximity to a target computing device in the environment and thereby establish an association of the target computing device with the modeled person, and

transmit the checkout list to a proxy interface device mapped to the target computing device; and

the proxy interface device communicatively, which is coupled to the computer vision monitoring platform and to the target computing device, and configured to transmit at least a part of the modeled data to the target computing device upon receipt of the modeled data.

15. The system of claim 14, wherein the target computing device is a checkout station.

16. The method of claim 15, wherein the proxy interface device emulates a product entry device.

17. The system of claim 16, wherein, the proxy interface device further comprises a product entry module communicatively coupled to a product entry device; and wherein the proxy interface device is further configured to communicate item data received through to the checkout station.

18. The system of claim 17, wherein the proxy interface device is further configured to communicate item data received through the product entry module to the computing platform.

19. The system of claim 14, wherein proxy interface device further comprises a user feedback system configured to output state of a checkout list.

20. The system of claim 19, wherein the user feedback system comprises of an audio system and graphical display system.