TWI787536B - Systems and methods to check-in shoppers in a cashier-less store - Google Patents

Abstract

Systems and techniques are provided for linking subjects in an area of real space with user accounts. The user accounts are linked with client applications executable on mobile computing devices. A plurality of cameras are disposed above the area. The cameras in the plurality of cameras produce respective sequences of images in corresponding fields of view in the real space. A processing system is coupled to the plurality of cameras. The processing system includes logic to determine locations of subjects represented in the images. The processing system further includes logic to match the identified subjects with user accounts by identifying locations of the mobile computing devices executing client applications in the area of real space and matching locations of the mobile computing devices with locations of the subjects.

Description

System and method for checking in shoppers in a cashierless store

This application petitions U.S. Provisional Patent Application No. 62/703,785 (Attorney Docket No. STCG 1006-1), filed July 26, 2018, and U.S. Nonprovisional Application No. 16, filed January 23, 2019 /255,573 (Attorney Docket No. STCG 1009-1), U.S. nonprovisional application Ser. Docket No. STCG 1005-1), a continuation-in-part of U.S. Patent Application No. 15/945,473, filed February 27, 2018, U.S. Patent Application No. 15/907,112 (Attorney Docket No. STCG 1002-1) (now U.S. Patent No. 10,133,933 published on November 20, 2018), the U.S. patent application No. 15/907,112 is a U.S. patent application filed on December 19, 2017 Continuation-in-Part of Ser. No. 15/847,796 (Attorney Docket No. STCG 1001-1), now U.S. Patent No. 10,055,853 issued Aug. 21, requesting U.S. Provisional Patent filed Aug. 07, 2017 Priority to Application No. 62/542,077 (Attorney's Case No. STCG 1000-1), which applications are hereby incorporated by reference.

This case relates to a system for associating a subject in a region of real space with a user account associated with a client application executing on a mobile computing device.

Explanation of related previous cases

Many technical challenges arise in identifying a subject within a region of real space, such as a person in a shopping store, uniquely associating an identified subject with a real person, or associating an authentication account with a responsible party. For example, consider such an image processing system deployed in a shopping store with multiple customers moving in the aisles between shelves and open spaces within the shopping store. Customers take items from the shelves and place them in their respective shopping carts or baskets. Customers can also place items on the shelf if they do not want the item. While the system can identify the subject in the image and the item the subject is taking, the system must accurately identify the real user account responsible for the item being taken by the subject.

In some systems, facial recognition or their biometric recognition techniques may be used to identify subjects in images and link them to accounts. However, this approach requires access by the image processing system to a database storing personal identifying biometric information (which is linked to an account). This is undesirable from a security and privacy standpoint in many settings.

It would be desirable to provide a system that more efficiently and automatically connects agents in regions of real space to users who are known to the system used to provide services to the agents. It would also be desirable to provide an image processing system that uses a large volume to identify a subject without requiring personally identifying biometric information from the subject.

A system and method for an operating system is provided for associating subjects, such as individuals in regions of real space, with user accounts. The system can use image processing to identify subjects in regions of real space without requiring personally identifying biometric information. A user account is associated with a client application executable on the mobile computing device. This function of linking an identified subject to a user account via image and signal processing presents complex problems of computational engineering regarding the type of image and signal data to be processed, what image and signal data processing is to be performed and How to decide actions from image and signal data with high reliability.

A system and method are provided for linking a subject in a region of real space with a user account. A user account is associated with a client application executable on the mobile computing device. A plurality of cameras or other sensors generate respective image sequences in corresponding fields of view in real space. Using these sequences of images, systems and methods are described for determining the locations of identified subjects represented in the images, and matching identified subjects with User account and match the location of the mobile device with the location of the subject.

In one embodiment described herein, a mobile device emits a signal that can be used to indicate the location of the mobile device in an area of real space. The system matches identified subjects with user accounts by using the emitted signals to identify the location of the mobile device.

In one embodiment, the signal transmitted by the mobile device includes images. In the illustrated embodiment, the client application on the mobile device causes the display on the mobile device to display a semaphore image that can be as simple as a specific color in an area of real space. The system matches identified subjects to user accounts by identifying the location of the mobile device by using an image recognition engine that determines the location of the mobile device displaying the sign image. The system includes sets of signal images. The system accepts a login communication from the client application on the mobile device identifying the user account prior to matching the user account to a subject identified in the real-space region. After accepting the login communication, the system sends the selected sign image from the set of sign images to the client application on the mobile device. The system sets the status of the selected sign image to assigned. The system receives displayed images of selected sign images, identifies the displayed images, and matches the identified images with designated images from the set of sign images. The system matches the location of the mobile device displaying the image of the identified sign located in the area of real space with the subject not yet linked to the identification. After matching the user account to the identified subject, the system sets the status of the identified sign image to available.

In one embodiment, the signal transmitted by the mobile device includes a radio frequency signal indicating the service location of the mobile device. The system receives location data sent by the client application on the mobile device. The system uses location data transmitted from mobile devices to match identified subjects with user accounts. The system uses location data transmitted from mobile devices from a plurality of locations over time intervals in an area of real space to match identifiers to user accounts. The step of matching the identified unmatched subject with a user account of a client application executing on the mobile device includes determining that all other mobile devices transmitting location information for the unmatched user account are separated from the mobile device by a predetermined distance, and The nearest unmatched identified subject for the mobile device is determined.

In one embodiment, the signal transmitted by the mobile device includes a radio frequency signal indicating the acceleration or orientation of the mobile device. In one embodiment, such acceleration data is generated by an accelerometer of the mobile computing device. In another embodiment, in addition to accelerometer data, orientation data from a compass on the mobile device is also received by the processing system. The system receives accelerometer data from the client application on the mobile device. The system uses accelerometer data transmitted from the mobile device to match identified subjects with user accounts. In this embodiment, the system uses accelerometer data transmitted from a mobile device from a plurality of locations over a time interval in a region of real space and indicates the location of an identified subject over the time interval in a region of real space. Derivative of data used to match identified subjects with user accounts.

In one embodiment, the system matches identified subjects with user accounts using a network trained to identify the location of the mobile device in an area of real space based on signals emitted by the mobile device. In such an embodiment, the signal transmitted by the mobile device includes location data and accelerometer data.

In one embodiment, the system includes a login data structure including a series of inventory items for an identified subject. The system associates the login data structure for the matched identified principal to the user account for the identified principal.

In one embodiment, the system processes payment for the series of inventory items for the identified subject from the payment method identified in the user account linked to the identified subject.

In one embodiment, the system matches identified subjects to user accounts without using personally identifying biometric information associated with the user accounts.

A method and a computer program product executable by a computer system are also described herein.

Other aspects and benefits of the present invention can also be found in the inspection of the subsequent drawings, detailed description of the invention and claims.

The following description is presented to enable any person of ordinary skill in the art to make and use the invention, and is presented in the context of a specific application and its elements. Various modifications to the disclosed embodiments will be apparent to those having ordinary knowledge in the art, and the general principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of the invention . Thus, the present invention is not intended to be limited to the embodiments shown, but is to be accorded the widest scope consistent with the principles and features disclosed herein. System overview

The system and various implementations of the subject technology are described with reference to FIGS. 1-11 . Referring to FIG. 1 to illustrate the system and process, an architectural level schematic diagram of the implemented system. Since FIG. 1 is an architectural diagram, certain details are omitted to improve clarity of description of the present invention.

The discussion lines of Figure 1 are organized as follows. First, the elements of the system are described, followed by their interconnections. Next, the use of the elements in the system is described in more detail.

FIG. 1 provides a block diagram level illustration of a system 100 . The system 100 includes a camera 114, a network node controlling image recognition engines 112a, 112b, and 112n, a subject tracking engine 110 deployed in the network node 102 (or nodes) on the network, and mobile computing devices 118a, 118b, 118m (collectively referred to as mobile computing devices 120), training database 130, subject database 140, user account database 150, image database 160, matching engine 170 (also known as processing platform) deployed in network nodes or nodes ) 103 and a communication network or network 181. The network node can only host one image recognition engine or several image recognition engines. The system can also include an inventory database and other supporting information.

As used herein, a network node is an addressable hardware device or virtual device attached to a network and capable of sending, receiving, or forwarding information over a communication channel to or from other network nodes Nodes send, receive or relay information. Examples of electronic devices that can be deployed as hardware network nodes include all kinds of computers, workstations, laptops, handheld computers, and smart phones. Network nodes can be implemented on cloud-based server systems. More than one virtual device configured as a network node can be implemented using a single physical device.

For the sake of simplicity, only three network nodes mastering the image recognition engine are shown in the system 100 . However, any number of network nodes hosting the image recognition engine can be connected to the subject tracking engine 110 through the network 181 . Likewise, three mobile computing devices are shown in system 100 . Likewise, any number of mobile computing devices can be connected to the network node 103 hosting the matching engine 170 through the network 181 . Likewise, the image recognition engine, subject tracking engine, matching engine, or other processing engine described herein may be implemented in a distributed architecture using more than one network node.

The interconnection of the elements of system 100 will now be described. The network 181 respectively connects the network nodes 101a, 101b and 101n, the image recognition engines 112a, 112b and 112n, the network node 102 which controls the subject tracking engine 110, the mobile computing devices 118a, 118b and 118m, and the training database 130 , the subject database 140 , the user account database 150 , the image database 160 and the network node 103 hosting the matching engine 170 are coupled. The camera 114 is connected to the subject tracking engine 110 through a network node hosting the image recognition engines 112a, 112b, and 112n. In one embodiment, the cameras 114 are installed in a shopping store such that groups of cameras 114 (two or more) with overlapping fields of view are positioned above each aisle to capture images of the real space in the store . In FIG. 1 , two cameras are arranged on the walkway 116a, two cameras are arranged on the walkway 116b, and three cameras are arranged on the walkway 116n. Cameras 114 are mounted above the walkway with overlapping fields of view. In such an embodiment, the cameras are configured with the customer moving through the shopping store and objects appearing in the fields of view of more than two cameras at any one time.

The cameras 114 may be temporally synchronized with each other such that imagery is captured at the same time or close in time and at the same image capture rate. The camera 114 may send respective continuous streams of images at a predetermined rate to the network nodes hosting the image recognition engines 112a-112n. Images captured in all cameras at the same time or close in time to an area covering real space are performed in the sense that the synchronized images can be recognized in the processing engine as representing different viewpoints of a subject with a fixed position in real space Synchronize. For example, in one embodiment, the camera sends image frames at a rate of 30 frames per second (fps) to respective network nodes that host the image recognition engines 112a-112n. Each frame together with the image data has a time stamp (timestamp), a camera identification (abbreviated as "camera_id"), and a frame identification (abbreviated as "frame_id"). Other embodiments of the disclosed technology may use different types of sensors, such as infrared or RF image sensors, ultrasonic sensors, thermal sensors, Lidar, etc., to generate this data. Multiple types of sensors may be used including, for example, ultrasonic or RF sensors in addition to the camera 114 producing RGB color output. Multiple sensors may be synchronized in time with each other such that frames are captured by the sensors at the same time or in close proximity in time and at the same image capture rate. In all of the embodiments described herein, sensors other than cameras, or multiple types of sensors, may be used to generate the sequence of images utilized.

The cameras installed on the walkways are connected to the respective image recognition engines. For example, in FIG. 1, two cameras installed on the walkway 116a are connected to the network node 101a that hosts the image recognition engine 112a. Similarly, the two cameras installed on the walkway 116b are connected to the network node 101b that hosts the image recognition engine 112b. Each image recognition engine 112a-112n of the host network node or nodes 101a-101n separately processes image frames received from a camera in each of the illustrated examples.

In one embodiment, each of the image recognition engines 112a, 112b, and 112n is implemented as a deep learning algorithm, such as a convolutional neural network (CNN (convolutional neural network) for short). In such embodiments, the CNN is trained using the training database 130 . In the embodiments described herein, image recognition of subjects in real space is based on identifying and grouping joints identifiable in the images, where the grouping of joints is attributable to individual subjects. For this joint-based analysis, the training database 130 has a large collection of images for each of the different types of joints of the subject. In an exemplary embodiment of a shopping store, the subjects are customers moving in the aisles between the shelves. In the exemplary embodiment, during the training of a CNN, the system 100 is referred to as a "training system". After training the CNN using the training database 130, the CNN is switched to production mode to process images of customers in a shopping store in real time.

In an exemplary embodiment, during production, system 100 is referred to as a runtime system (also referred to as an inference system). The CNN in each image recognition engine generates the joint data structure for the image in its stream of respective images. As in the embodiment described herein, an array of joint data structures is generated for each processed image such that each image recognition engine 112a-112n generates an output stream of the array of joint data structures. The arrays of joint data from cameras with overlapping views are further processed to form groups of joints and to identify such groups of joints as subjects. These groups of joints may not uniquely identify the individual in the image or an authentic user account for the individual in the image, but can be used to track subjects in the area. Subjects can be identified and tracked by the system during their presence in regions of real space using the identifier "subject_id".

For example, when a customer enters a shopping store, the system uses joint analysis as described above to identify the customer and is assigned a "subject_id". However, an identifier is not linked to a subject's real-world identity, such as a user account, name, driver's license, email address, mailing address, credit card number, bank account number, driver's license number, etc., or to identify biometrics Authentication (biometric identification), such as fingerprints (finger prints), facial recognition, hand geometry, retina scan, iris scan, voice recognition, etc. Therefore, identified subjects are anonymous. Details of an example technique for subject authentication and tracking are set forth in U.S. Patent No. 10,055,853, issued August 21, 2018, entitled "Subject Identification and Tracking Using Image Recognition Engine," which is hereby incorporated by reference as if Exactly the same as proposed here.

In this example, subject tracking engine 110 hosted on network node 102 receives a continuous stream of arrays of joint data structures for subjects from image recognition engines 112a-112n. The subject tracking engine 110 processes the array of joint data structures and translates coordinates of elements in the array of joint data structures corresponding to images in different sequences into candidate joints with coordinates in real space. For each set of synchronized images, for analogy purposes, the combination of candidate joints identified throughout the real space can be considered like a galaxy of candidate joints. For each subsequent point in time, the actions of the candidate joints causing the galaxy to change over time are recorded. The output of subject tracking engine 110 is stored in subject database 140 .

The subject tracking engine 110 uses logic to identify groups or groups of candidate joints having coordinates in real space as subjects in real space. For analogy purposes, each set of candidate points is like a constellation of candidate joints at various points in time. The constellation of candidate joints can be shifted over time.

In an example embodiment, the logic to identify the set of candidate joints includes a heuristic function based on physical relationships between joints of the subject in real space. These heuristic functions are used to identify groups of candidate joints as subjects. The set of candidate joints includes individual candidate joints that have relationships to other individual candidate joints according to heuristic parameters and a subset of candidate joints in a given group/collection that have been identified or can be identified as individual subjects.

In the example of a shopping store, the system processes payment for items purchased by the customer when the customer finishes shopping and moves out of the store. In a cashierless store, the system must link the customer to a "user account" that contains the preferred payment method provided by the customer.

As noted above, an "identified subject" is anonymous because information about joints and relationships between joints is not stored as biometric information linked to an individual or user account.

The system includes a matching engine 170 (hosted on the network node 103) for processing signals received from the mobile computing device 120 (onboard the subject) to match identified subjects with user accounts. Matching can be performed by identifying the location of the mobile device executing the client application in an area of real space (e.g., a shopping store) and matching the location of the mobile device to the location of the subject without using personally identifying biometric information from the imagery .

The actual communication path through network 181 to network node 103 hosting matching engine 170 may be point-to-point over public and/or private networks. Communications can take place over various networks 181, such as private networks, VPNs, MPLS circuits, or the Internet, and can use appropriate application programming interfaces (APIs; application programming interfaces) and data exchange formats, such as presentation layer status Conversion (REST; Representational State Transfer), JavaScript ^TM Object Notation (JSON; JavaScript ^TM Object Notation), Extensible Markup Language (XML; Extensible Markup Language), Simple Object Access Protocol (SOAP; Simple Object Access Protocol), Java ^TM Message Service (JMS; Java ^TM Message Service) and/or the Java Platform Module System. All communication can be encrypted. Communication is generally over a network such as LAN (Local Area Network), WAN (Wide Area Network), telephone network (Public Switched Telephone Network (PSTN; Public Switched Telephone Network )), session initiation protocol (SIP; Session Initiation Protocol), wireless network, point-to-point network, star network, token ring network, hub network, Internet, including via Mobile Internet of protocols such as EDGE, 3G, 4G LTE, Wi-Fi and WiMAX. Additionally, various authorization and authentication techniques, such as username/password, Open Authorization (OAuth; Open Authorization), Kerberos, SecureID, digital certificates, and more, can be used to secure communications.

The techniques disclosed herein can be implemented in the context of any computer implementation system, including database systems, multi-tenant environments, or relational database implementations (such as Oracle™ compatible database implementations, IBM DB2 Enterprise Server™ compatible relational database implementation, MySQL™ or PostgreSQL™ compatible relational database implementation, or Microsoft SQL Server™ compatible relational database implementation), or a NoSQL™ non-relational database implementation (such as Vampire™ Compatible non-relational database implementation, Apache Cassandra™ compatible non-relational database implementation, BigTable™ compatible non-relational database implementation, or HBase™ or DynamoDB™ compatible non-relational database implementation). Furthermore, the disclosed techniques can be implemented using incompatible programming models such as MapReduce™, bulk synchronous programming, MPI primitives, etc., or different scalable batch and stream management Systems such as Apache Storm™, Apache Spark™, Apache Kafka™, Apache Flink™, Truviso™, Amazon Elasticsearch Service™, Amazon Web Services™ (AWS), IBM Info-Sphere™, Borealis™, and Yahoo! S4™ . Camera layout

The camera 114 is arranged to track a multi-joint body (or entity) in a three-dimensional (abbreviated as 3D) real space. In an example embodiment of a shopping store, the physical space may include an area of the shopping store where items for sale are stacked in shelves. A point in real space can be represented by an (x, y, z) coordinate system. Each point in the region of real space for which the system is deployed is covered by the fields of view of two or more cameras 114 .

In a shopping store, shelves and other inventory display structures can be arranged in various ways, such as along the walls of the shopping store, or in rows forming aisles, or a combination of the two. FIG. 2 shows the arrangement of shelves forming the aisle 116a viewed from one end of the aisle 116a. Two cameras, Camera A 206 and Camera B 208, are positioned above the aisle 116a at a predetermined distance from the roof 230 and floor 220 of the shopping store on an inventory display structure such as a shelf. Cameras 114 include cameras disposed over and having their fields of view over respective portions of the inventory display structure and floor area in real space. The coordinates in real space of the members of the set of candidate joints (identified as subjects) identify the positions of the subjects in the floor area. In FIG. 2, body 240 is holding mobile computing device 118a and standing on floor 220 in walkway 116a. The mobile computing device can send and receive signals through the wireless network 181 . In one embodiment, the mobile computing device 120 is connected via a wireless access point through a wireless network using, for example, the Wi-Fi protocol or other wireless protocols such as Bluetooth, ultra-wideband, and Zigbee. (WAP; wireless access point) 250 and 252 for communication.

In the example embodiment of a shopping store, the real space may include the entirety of the floor 220 in the shopping store from which inventory can be accessed. The cameras 114 are positioned and oriented such that the area of the floor 220 and the shelves are visible to at least two cameras. Camera 114 also covers shelves 202 and 204 and at least a portion of the floor space in front of shelves 202 and 204 . The camera angles are selected to both have a steep perspective, straight down and give a fuller angled perspective than the customer's body image. In an exemplary embodiment, the cameras 114 are configured at eight (8) feet high or higher throughout the shopping store.

In FIG. 2, cameras 206 and 208 have overlapping fields of view covering the space between shelf A 202 and shelf B 204 having overlapping fields of view 216 and 218, respectively. A position in real space is represented as an (x, y, z) point of the real space coordinate system. "x" and "y" represent locations on a two-dimensional (2D) plane, which may be the floor 220 of a shopping store. The value "z" is the height of a point above the 2D plane at floor 220 in a configuration.

FIG. 3 shows the walkway 116a viewed from the top of FIG. 2, further illustrating an example arrangement of the locations of the cameras 206 and 208 above the walkway 116a. Cameras 206 and 208 are located closer to opposite ends of walkway 116a. Camera A 206 is positioned at a predetermined distance from shelf A 202 and camera B 208 is positioned at a predetermined distance from shelf B 204 . In another embodiment where more than two cameras are positioned above the walkway, the cameras are positioned equidistant from each other. In such an embodiment, two cameras are located near opposite ends and a third camera is located in the middle of the aisle. It is to be appreciated that several camera arrangements are possible. Joint Data Structure

The image recognition engines 112a-112n receive a sequence of images from the camera 114 and process the images to generate an array of corresponding joint data structures. In one embodiment, the image recognition engines 112a-112n identify one of 19 possible joints for each subject at each element of the image. The possible joints can be grouped in two categories: foot joints and non-foot joints. Type 19 joint classification is used for all non-joint features of the subject (ie, elements of the image not classified as joints). Foot joints: Ankle (left and right) Non-foot joints: neck nose eye (left and right) ear (left and right) shoulders (left and right) elbow (left and right) Wrist (left and right) hip (left and right) knee (left and right) non-joint

The array of joint data structures for a particular image categorizes elements of a particular image by joint type, time of the particular image, and coordinates of the element within the particular image. In one embodiment, the image recognition engines 112a~112n are convolutional neural networks (CNN), the joint type is one of the 19 types of subjects, and the time of the specific image is generated by the source camera 114 for the specific image The time stamp of the image, and the coordinates (x, y) identify the position of the component on the 2D image plane.

The output of the CNN is a matrix of confidence arrays per camera for each image. The matrix of confidence arrays is transformed into a matrix of joint data structures. A joint data structure 400 as shown in FIG. 4 is used to store information about each joint. The joint data structure 400 identifies the x and y positions of elements in a particular image in the 2D image space of the camera from which the image is received. The joint number identifies the type of joint identified. For example, in one embodiment, the values range from 1 to 19. A value of 1 indicates the joint is the left ankle, a value of 2 indicates the joint is the right ankle, and so on. The type of joint is selected using the confidence array for the elements in the CNN's in-out matrix. For example, in one embodiment, if the value corresponding to the left ankle is the highest in the confidence array for that image element, then the value of the next joint number is "1".

The confidence number indicates how confident the CNN is in predicting this joint. If the value of the confidence number is high, it means that the CNN is confident in its predictions. An integer-Id is specified in the joint data structure to uniquely identify it. Following the above mapping, the output matrix of confidence arrays for each image is transformed into an array of joint data structures for each image. In one embodiment, joint analysis includes performing k-nearest neighbor union, Gaussian mixture, and morphology transformation of various images on each input image. The result contains an array of joint data structures that can be stored as bit masks in a ring buffer that maps image numbers to bit masks at various points in time. subject tracking engine

主體資料結構Tracking engine 110 is configured to receive an array of joint data structures generated by image recognition engines 112a-112n corresponding to images of a sequence of images from cameras with overlapping views. The array of joint data structures for each image is sent by the image recognition engines 112 a - 112 n to the tracking engine 110 via the network 181 . The tracking engine 110 translates coordinates corresponding to elements in the array of joint data structures of images in different sequences into candidate joints with coordinates in real space. Tracking engine 110 includes logic to identify a set of candidate joints (constellation of joints) with coordinates in real space as subjects in real space. In one embodiment, the tracking engine 110 accumulates an array of joint data structures from the image recognition engine for all cameras at a given moment in time and stores this information as a dictionary in the subject database 140 for use in Constellation diagrams for identifying candidate joints. Dictionaries can be arranged in key-value pairs, where the key is the camera id and the value is an array from the camera's joint data structure. In such embodiments, this dictionary is used in a heuristics-based analysis to determine candidate joints and to assign joints to subjects. In such an embodiment, the high-level input, processing, and output of the tracking engine 110 are illustrated in Table 1. Details of the logic applied by the subject tracking engine 110 to create a subject by combining candidate joints and to track the subject's movement in a region of real space are published on August 21, 2018 entitled "Subject Identification and Tracking US Patent No. 10,055,853, "Using Image Recognition Engine," which is hereby incorporated by reference. Table 1: Input, processing and output from subject tracking engine 110 in an example embodiment.

Master data structure

The subject tracking engine 110 uses heuristics to connect the joints of the subjects identified by the image recognition engines 112a-112n. In doing so, the subject tracking engine 110 creates new subjects and updates the positions of existing subjects by updating their respective joint positions. The subject tracking engine 110 uses triangulation techniques to project joint positions from 2D space coordinates (x, y) to 3D real space coordinates (x, y, z). FIG. 5 illustrates a principal data structure 500 used to store principals. The subject data structure 500 stores subject-related data as a key-value dictionary. The key is frame_number and the value is another key-value dictionary where the key is camera_id (camera identification) and the value is a series of 18 joints (of the subject) with their positions in real space. The subject data is stored in the subject database 140 . Each new subject is also assigned a unique identifier, which is used to access the subject's data in subject database 140 .

In one embodiment, the system identifies the joints of the subject and creates a skeleton of the subject. A skeleton is projected into real space, which indicates the position and orientation of the subject in real space. This is called "pose estimation" in the field of machine vision. In one embodiment, the system displays the orientation and position of the subject in real space on a graphical user interface (GUI). In one embodiment, the image analysis is anonymous, ie, the unique identifier assigned to the joint analysis does not personally identify the subject as described above. matching engine

Matching engine 170 includes logic to match identified subjects with their respective user accounts by identifying the locations of mobile devices (carried by identified subjects) executing client applications in regions of real space. In one embodiment, the matching engine uses a variety of techniques, independently or in combination, to match identified subjects to user accounts. The system can be implemented without retaining biometric information about users such that biometric information about account holders is not exposed to the security and privacy concerns that arise from the distribution of such information.

In one embodiment, a customer, upon entering a shopping store, uses a client application executing on a personal mobile computing device, identifying a real user account to be associated with the client application on the mobile device to log into the system. The system then sends the selected "sign" image from the group of unassigned sign images in the image database 160 to the client application executing on the mobile device. When the same image has not been released for use with another client application in the store, the sign image is unique to that client application in the shopping store until the system has matched the user account to the identifying of the subject. After this matching, the sign image becomes available for reuse. The client application causes the mobile device to display a beacon image, which is displayed as a signal emitted by the mobile device to be detected by the system. The matching engine 170 uses the image recognition engines 112a-n or a separate image recognition engine (not shown in FIG. 1 ) to recognize the sign image and determine the location of the mobile computing device displaying the sign in the shopping store. The matching engine 170 matches the location of the mobile computing device to the location of the identified subject. Matching engine 170 then links the identified subject (stored in subject database 140 ) to the user account (stored in user account database 150 ) that is linked to the client application during the subject's presence in the shopping store. No biometric information is used to match identified subjects to user accounts, and nothing is stored to support this process. That is, no information in the sequence of images is used to compare with stored biometric information for the purpose of matching identified subjects with user accounts in support of this process.

In other embodiments, the matching engine 170 alternatively or in combination uses other signals from the mobile computing device 120 to link identified subjects to user accounts. Examples of such signals include a service location signal identifying the location of the mobile computing device in an area of real space, the speed and orientation of the mobile computing device obtained from an accelerometer, the compass of the mobile computing device, and the like.

In some embodiments, although embodiments are provided that do not hold any biometric information about account holders, the system may use biometric information to assist in matching identified subjects that have not been linked to user accounts. For example, in one embodiment, the system stores a customer's "hair color" in his or her user record. During the matching process, the system may use, for example, the subject's hair color as an additional input to disambiguate and match the subject to the user account. If the user has red dyed hair and only one subject with red dyed hair is in the area of real space or in close proximity to the mobile computing device, then the system may select a subject with red hair color to match the user account.

The flowcharts in FIGS. 6 through 9C , individually or in combination, present the process steps of four techniques that may be used by the matching engine 170 . Signal image

FIG. 6 is a flowchart 600 presenting process steps for a first technique for matching subjects identified in regions of real space with their respective user accounts. In the example of a shopping store, the subjects are customers (or shoppers) moving in the aisles between shelves and other open spaces in the store. The process begins at step 602 . When the subject enters an area of real space, the subject opens the client application on the mobile computing device and attempts to log in. At step 604 the system verifies user credentials (eg, by querying the user account database 150 ) and accepts login communications from the client application for associating the authenticated user account with the mobile computing device. The system has determined that the user account for the client application has not been associated with the identified principal. At step 606, the system sends the sign image to the client application for display on the mobile computing device. Examples of sign images include various shapes of solid colors, such as red rectangles, pink elephants, and so on. Various images can be used as markers, preferably suitable for high confidence recognition by an image recognition engine. Each sign image may have a unique identifier. The processing system includes a method for receiving a login communication from a client application on a mobile device that identifies a user account prior to matching the user account to a subject identified in the region of real space, and after accepting the login communication, from the group number Logic to send selected ones of the logo images to the client application on the mobile device.

In one embodiment, the system selects available sign images from the image database 160 for sending to the client application. After sending the sign image to the client application, the system changes the status of the sign image in the image database 160 to "assigned" so that the image is not assigned to any other client application. The status of the image remains "assigned" until the process to match the identified subject to the mobile computing device is complete. After matching is done, the state can be changed to "Available". This allows the use of round robin for small groups of signs in a given system, simplifying the image recognition problem.

The client application receives the sign image and displays it on the mobile computing device. In one embodiment, the client application also increases the brightness of the display to increase the visibility of the image. Imagery is captured by one or more cameras 114 and sent to an image processing engine called WhatCNN. At step 608 the system uses WhatCNN to recognize sign images displayed on the mobile computing device. In one embodiment, WhatCNN is a Convolutional Neural Network trained to process bounding boxes defined in images to generate a classification of the identified subject's hand. A WhatCNN is trained on image frames from a camera. In the example embodiment of the shopping store, for each hand joint in each image frame, WhatCNN identifies whether the hand joint is empty or not. WhatCNN also recognizes the symbol image identifier (in the image database 160) or the SKU (stock keeping unit) number of the item in the hand joint, indicating that the confidence value of the item in the hand joint is false SKU items (i.e., which are not part of the shopping store inventory) and context of hand joint positions in the image frame.

As mentioned above, two or more cameras with overlapping fields of view capture images of subjects in real space. The joints of a single subject can be displayed in image frames of multiple cameras in separate image channels. The per-camera WhatCNN model recognizes the image of a sign (displayed on a mobile computing device) in the subject's hand (represented by hand joints). The coordination logic incorporates the output of the WhatCNN model into the merged data structure enumeration identifiers of the identified subject's left hand (called left_hand_classid) and right hand (right_hand_classid) mid-sign images (step 610). The system stores this information in a dictionary that maps subject_id along with timestamps (including the joint's position in real space) to left_hand_classid and right_hand_classid. Details of WhatCNN are set forth in US Patent Application Serial No. 15/907,112, filed February 27, 2018, entitled "Item Put and Take Detection Using Image Recognition," which is hereby incorporated by reference as if fully set forth herein.

At step 612, the system checks whether the sign image sent to the client application was recognized by WhatCNN by iterating the output of the WhatCNN model for all identified hands of the subject. If the sign image is not recognized, at step 614 the system sends a reminder to the client application to display the sign image on the mobile computing device and repeats process steps 608 to 612 . Otherwise, if the sign image is recognized by WhatCNN, the system matches the user_account (user account) associated with the client application (from the user account database 150) to the subject_id (subject_id) of the identified subject holding the mobile computing device identity) (from subject database 140) (step 616). In one embodiment, the system maintains this mapping (subject_id-user_account) until the subject appears in a region of real space. The process ends at step 618. service location

Flowchart 700 in FIG. 7 presents process steps for a second technique for matching identified principals to user accounts. This technology uses radio frequency signals emitted by mobile devices, which indicate the location of the mobile device. The process begins at step 702, and as described above at step 604, the system accepts a login communication from a client application on the mobile computing device for linking an authenticated user account to the mobile computing device. In step 706, the system receives service location information from mobile devices in the real space area at regular intervals. In one embodiment, the system determines location using latitude and longitude coordinates of the mobile computing device transmitted from a global positioning system (GPS) receiver of the mobile computing device. In one embodiment, the service location of the mobile computing device obtained from GPS coordinates has an accuracy between 1 and 3 meters. In another embodiment, the service location of the mobile computing device obtained from GPS coordinates has an accuracy between 1 and 5 meters.

Other techniques can be used independently or in combination with the above techniques to determine the service location of the mobile computing device. Examples of such techniques include using signal strengths from different wireless access points (WAPs), such as 250 and 252 shown in FIGS. 2 and 3, as a measure of how far the mobile computing device is from the respective access points instructions. The system then uses the known locations of Wireless Access Points (WAPs) 250 and 252 to triangulate and determine the location of the mobile computing device in the area of real space. Other types of signals emitted by mobile computing devices, such as Bluetooth, ultra-wideband, and Zigbee, may also be used to determine the service location of the mobile computing device.

The system monitors the mobile device's service location at step 708 at regular intervals, such as every second, with client applications that are not yet connected to the identified subject. At step 708, the system determines the distance of the mobile computing device with unmatched user accounts from all other mobile computing devices with unmatched user accounts. The system compares this distance with a predetermined threshold distance "d" (say 3 meters). If the mobile computing device is at least "d" away from all other mobile devices with unmatched user accounts (step 710), the system determines the subject that has not recently been associated with a mobile computing device (step 714). At step 712, the location of the identified subject is obtained from the output of the JointsCNN. In one embodiment, the subject's location obtained from JointsCNN is more accurate than the mobile computing device's service location. In step 616, the system performs the same process as described above in flowchart 600 to match the subject_id of the identified subject with the user_account of the client application. The process ends at step 718.

No biometric information is used to match identified subjects to user accounts, and nothing is stored to support this process. That is, no information in the sequence of images is used to compare with stored biometric information for the purpose of matching identified subjects with user accounts in support of this process. Accordingly, the logic used to match an identified subject to a user account does not operate using personally identifying biometric information associated with the user account. speed and orientation

Flowchart 800 in FIG. 8 presents process steps for a third technique for matching identified principals to user accounts. This technique uses signals emitted by the accelerometer of the mobile computing device to match identified subjects with client applications. The process begins at step 802 . The process begins at step 604 by accepting a login communication from a client application as described above in the first and second techniques. At step 806, the system receives a signal transmitted from the mobile computing device carrying data from an accelerometer on the mobile computing device in a region of real space, which may be sent at regular intervals. At step 808, the system calculates the average speed of all mobile computing devices with unmatched user accounts.

The accelerometer provides the acceleration of the mobile computing device along three axes (x, y, z). In one embodiment, velocity is calculated by taking acceleration values at small time intervals (eg, every 10 milliseconds) to calculate the current velocity at time "t", i.e. v _t = v ₀ + a _t , where v ₀ is the initial velocity. In one embodiment, v ₀ is initialized to "0", and every time t+1, v _t becomes v ₀ . The velocities along the three axes are then combined to determine the overall velocity of the mobile computing device at time "t". Finally at step 808, the system calculates the average of the speeds of movement of all computing devices over a larger period of time, say 3 seconds, which is long enough for the average individual's walking gait, or exceeds longer period of time.

In step 810, the system calculates the Euclidean distance (also referred to as L2 ) between the velocities of all pairs of mobile computing devices with unmatched client applications for unlinked identified subjects. Norm (norm)). The velocities of the agents are derived from changes in the positions of their joints with respect to time, obtained from joint analysis and stored in separate agent data structures 500 with time stamps. In one embodiment, the location of the center of mass of each body is determined using joint analysis. The velocity or other derivative of the subject's centroid location data is used for comparison with the velocity of the mobile computing device. For each subject_id-user_account pair, if the value of the Euclidean distance between their respective speeds is less than threshold_0 (threshold_0), the score_counter (score count) of the subject_id-user_account pair is incremented. The above process is performed at regular intervals, so the score_counter for each subject_id-user_account pair is updated.

At regular intervals (eg, every second), the system compares the score_counter value for each unmatched user account in the pair to each identified subject that has not been linked (step 812). If the highest score is greater than threshold_1 (step 814), at step 816 the system calculates the difference between the highest score and the second highest score (for pairs of identical user accounts with different principals). If the difference is greater than threshold_2, at step 818 the system selects the mapping of user_account to identified principals and follows the same process as at step 616 described above. The process ends at step 820.

In another embodiment, when JointsCNN recognizes a hand holding a mobile computing device, the velocity of the hand (of the identified subject) holding the mobile computing device is used instead of the velocity of the subject's center of mass in the above process. This improves the performance of the matching algorithm. To determine the threshold values (threshold_0, threshold_1, threshold_2), the system uses training data with labels assigned to the images. During training, various combinations of thresholds are used and the output of the algorithm is matched to the ground truth labels of the images. The threshold value resulting in the best overall specified accuracy is for use in production (or inference).

No biometric information is used to match identified subjects to user accounts, and nothing is stored to support this process. That is, no information in the sequence of images is used to compare with stored biometric information for the purpose of matching identified subjects with user accounts in support of this process. Accordingly, the logic used to match an identified subject to a user account does not operate using personally identifying biometric information associated with the user account. network integration

Network integration is a learning paradigm where many networks are used jointly to solve problems. Integrating typically improves the predictive accuracy obtained from a single classifier by a factor of the effort and cost of validating and learning multiple models. In a fourth technique for matching user accounts to not-yet-linked identities, the second and third techniques presented above are used jointly in integration (or network integration). In order to use the two techniques in the integration, relevant features are extracted from the application of the two techniques. FIGS. 9A-9C are process steps for extracting features, training integration, and using the trained integration to predict matches of user accounts to not-yet-linked identified subjects (in flowchart 900)

FIG. 9A presents process steps for generating features using a second technique that uses a mobile computing device's service location. The process begins at step 902 . At step 904, Calculate Count_X for the second technique, indicating that the service location of the mobile computing device with the unmatched user account is X meters away from all other mobile computing devices with the unmatched user account number of times. In step 906, the Count_X values of all tuples of subject_id-user_account pairs are stored by the system for use by the integration. In one embodiment, multiple values of X are used, such as lm, 2m, 3m, 4m, 5m (steps 908 and 910). For each value of X, the count is stored as a dictionary mapping subject_id-user_account tuples to count scores, which are integers. In the example where a value of X of 5 is used, 5 such dictionaries are created at step 912 . The process ends at step 914.

Figure 9B presents process steps for generating features using a third technique that uses the speed of a mobile computing device. The process starts at step 920. At step 922, a Count_Y for the third technique is determined, which is equal to the score_counter value indicating the number of times the Euclidean distance between a particular subject_id-user_account pair is less than threshold_0. At step 924, the Count_Y values for all tuples of subject_id-user_account pairs are stored by the system for use by the integration. In one embodiment, multiple values of threshold_0 are used, eg, five different values (steps 926 and 928). Count_Y is stored as a dictionary mapping subject_id-user_account tuples to count scores (which are integers) for each value of threshold_0. In the example where a threshold value of 5 is used, 5 such dictionaries are created at step 930 . The process ends at step 932.

Features from the second and third techniques are then used to create a labeled training dataset and used to train the network ensemble. To collect such datasets, multiple subjects (shoppers) walk in an area of real space like a shopping store. Images of these subjects are collected using cameras 114 at regular time intervals. A human labeler looks at the images and assigns the correct identifiers (subject_id and user_account) to the images in the training data. This process is described in the flowchart 900 presented in Figure 9C. The process begins at step 940. In step 942, features in the form of Count_X and Count_Y dictionaries obtained from the second and third techniques are compared with the corresponding live labels specified by human labelers on the imagery to identify the correct subject_id and user_account Matches (true) and incorrect (false) matches.

As we only have two kinds of results for subject_id and user_account: true or false, a binary classifier is trained using this training data set (step 944). Common methods for binary classification include decision tree, random forest, neural network, gradient boost, support vector machine, etc. Use the trained binary classifier to classify new probabilistic observations as true or false. The trained binary classifier is used in production (or inference) by being given as input Count_X and Count_Y dictionaries for subject_id-user_account. At step 946, the trained binary classifier classifies each tuple as true or false. The process ends at step 948.

If there is an unmatched computing device in the area of the real space after applying the above four techniques, the system sends a notification to the mobile computing device to open the client application. If the user accepts the notification, the client application will display the sign image as described in the first technique. The system will then follow the steps in the first technique to sign the shopper in (matching subject_id to user_account). If the customer does not respond to the notification, the system will send a notification to an employee in the shopping store indicating the location of the unmatched customer. The employee can then walk up to the customer and ask him to open the client application on his mobile computing device to log into the system using the sign image.

No biometric information is used to match identified subjects to user accounts, and nothing is stored to support this process. That is, no information in the sequence of images is used to compare with stored biometric information for the purpose of matching identified subjects with user accounts in support of this process. Accordingly, the logic used to match an identified subject to a user account does not operate using personally identifying biometric information associated with the user account. architecture

The example architecture of the system of the four techniques presented above is applied to match user_accounts to unconnected principals in the region of real space presented in FIG. 10 . Since FIG. 10 is an architectural diagram, certain details are omitted to improve clarity of description of the present invention. The system presented in FIG. 10 receives image frames from a plurality of cameras 114 . As noted above, in one embodiment, the cameras 114 may be temporally synchronized with each other such that images are captured at the same time or closely in time and at the same image capture rate. Images captured in all cameras at the same time or temporally close to an area encompassing real space are synchronized images that can be recognized in the processing engine as representing different moments in time in real space of subjects with fixed positions Synchronize in the sense of viewpoint. The images are stored in a ring buffer of 1002 image frames per camera.

The "subject recognition" subsystem 1004 (also referred to as the first image processor) processes image frames received from the camera 114 to identify and track subjects in real space. The first image processor includes a subject image recognition engine, such as the aforementioned JointsCNN.

The "semantic diffing" subsystem 1006 (also referred to as the second image processor) includes a background image recognition engine that receives corresponding sequences of images from a plurality of cameras and semantically identifies differences that are significant in the background (i.e. , like the inventory display structure of a shelf), for example as they relate to placing and fetching inventory items over time in images from various cameras. The second image processor receives the output of the subject recognition subsystem 1004 and the image frame from the camera 114 as input. Details of the "Semantic Diffing" subsystem are contained in U.S. Patent Application Serial No. 15/945,466, filed April 04, 2018, entitled "Predicting Inventory Events using Semantic Diffing" and filed April 04, 2018, entitled "Predicting Inventory Events using Foreground/Background Processing" in US Patent Application No. 15/945,473, both of which are hereby incorporated by reference as if fully set forth herein. The second image processor processes the identified context changes to make a first set of detections of the inventory item being taken by the identified subject and the inventory item being placed on the inventory display structure by the identified subject. The first set of detections is also known as background detections for placing and taking inventory items. In the example of a shopping store, the first detection identifies inventory items taken from shelves or placed on shelves by customers or employees of the store. The semantic difference subsystem includes logic to associate identified context changes with identified subjects.

The "region proposal" subsystem 1008 (also referred to as the third image processor) includes a foreground image recognition engine that receives corresponding sequences of images from the plurality of cameras 114 and semantically identifies objects that are prominent in the foreground (i.e., shoppers, their hands and inventory items), such as when they place and take inventory items in the footage from various cameras over time. The region proposal subsystem 1008 also receives the output of the subject identification subsystem 1004 . A third image processor processes the sequence of images from the camera 114 to identify and classify foreground changes represented in the images in the corresponding sequence of images. The third image processor processes the identified foreground change to make a second set of detections of the inventory item being taken by the identified subject and the inventory item being placed on the inventory display structure by the identified subject. The second set of detections is also known as background detections for putting and taking inventory items. In the example of a shopping store, the second set of detections identifies taking inventory items and placing inventory items on the inventory display structure by customers and employees of the store. Details of the region proposal subsystem are set forth in U.S. Patent Application Serial No. 15/907,112, filed February 27, 2018, entitled "Item Put and Take Detection Using Image Recognition," which is hereby incorporated by reference as if fully set forth herein. Presented in general.

The system depicted in FIG. 10 includes selection logic 1010 to process the first and second sets of detections to generate a log data structure that includes a series of inventory items for identified subjects. The selection logic 1010 selects an output from either the semantic difference subsystem 1006 or the region proposal subsystem 1008 for take or place in real space. In one embodiment, the selection logic 1010 uses the confidence scores produced by the semantic difference subsystem for the first set of detections and the confidence scores produced by the region proposal subsystem for the second set of detections to make choose. The output of the subsystem with the higher confidence score for a particular detection is selected and used to generate a log data structure 1012 (also known as a cart data structure) comprising a series of inventory items associated with the identified subject (and their number).

To process payments for items in the log data structure 1012, the system in FIG. 10 applies four techniques for matching identified principals (associated with log data) to user_accounts, which include Is the payment method for credit or bank account information. In one embodiment, the four techniques are applied sequentially as depicted in the figure. If the process steps in flowchart 600 for the first technique generate a match between the principal and the user account, then this information is used by the payment processor 1036 to check for the inventory item in the log data structure Charge customers. Otherwise (step 1028 ), the process steps set forth in flowchart 700 for the second technique are followed and the user account is used by payment processor 1036 . If the second technique fails to match the user account to the principal (1030), then the process steps set forth in flowchart 800 for the third technique follow. If the third technique is unable to match the user account to the principal (1032), then follow the process steps set forth in flowchart 900 for the fourth technique to match the user account to the principal.

If the fourth technique fails to match the user account with the principal (1034), the system sends a notification to the mobile computing device to open the client application and follows the steps set forth in flowchart 600 for the first technique. If the customer does not respond to the notification, the system will send a notification to an employee in the shopping store indicating the location of the unmatched customer. The employee can then walk up to the customer and ask him to open the client application on his mobile computing device to log into the system using the sign image (step 1040). It is to be appreciated that in other embodiments of the architecture presented in FIG. 10, fewer than four techniques can be used to match user accounts to not-yet-linked identities. network structure

FIG. 11 presents the architecture of a network hosting matching engines 170 hosted on network nodes 103 . In the illustrated embodiment, the system includes a plurality of network nodes 103 , 101 a - 101 n and 102 . In such embodiments, network nodes are also referred to as processing platforms. Processing platforms (network nodes) 103 , 101 a - 101 n and 102 and cameras 1112 , 1114 , 1116 , . . . 1118 are connected to network 1181 .

FIG. 11 shows a plurality of cameras 1112, 1114, 1116, . . . 1118 connected to a network. A large number of cameras can be deployed in a given system. In one embodiment, the cameras 1112 to 1118 are connected to the network 1181 using Ethernet-based connectors 1122 , 1124 , 1126 and 1128 respectively. In such embodiments, the Ethernet connector has a data transmission speed of 1 gigabit per second, also known as Gigabit Ethernet. It is to be appreciated that in other embodiments, the camera 114 is connected to the network using other types of network connections that can have faster or slower data transfer rates than Gigabit Ethernet. Also, in alternative embodiments, groups of cameras may be directly connected to individual processing platforms, and the processing platforms may be coupled to a network.

Storage subsystem 1130 stores the basic programming and data structures that provide the functionality of certain embodiments of the present invention. For example, various modules implementing the functional characteristics of matching engine 170 may be stored in storage subsystem 1130 . Storage subsystem 1130 is an example of computer readable memory, which includes non-transitory data storage media having computer instructions stored in memory executable by the computer to perform the data described herein All or any combination of image processing and image processing, the instructions include logic for associating subjects in regions of real space with user accounts, for determining the location of identified subjects represented in images, by The logic in regions of the space whereby the processes described herein are executed by the client application to identify the location of the mobile computing device to match the identified subject to a user account. In other embodiments, computer instructions may be stored in other types of memory, including portable memory, which includes a non-transitory data storage medium or media readable by a computer.

These software modules are generally executed by the processor subsystem 1150 . The master memory subsystem 1132 typically includes several memories, including a main random access memory (RAM; random access memory) 1134 for storing instructions and data during program execution, and a RAM for storing fixed instructions therein. Read-only memory (ROM; read-only memory) 1136 . In one embodiment, RAM 1134 is used as a buffer for storing subject_id-user_account tuples matched by matching engine 170 .

File storage subsystem 1140 provides persistent storage for program and data files. In the exemplary embodiment, the storage subsystem 1140 includes four 120 gigabyte (GB; Gigabyte) solid state drives (SSD; solid state) in a RAID 0 (Redundant Array of Independent Disks) 1142 arrangement identified by a number disks). In an exemplary embodiment, the user account data in the user account database 150 and the image data in the image database 160 that are not in RAM are stored in RAID 0. In an exemplary embodiment, the hard disk drive (HDD; hard disk drive) 1146 is slower in access speed than the RAID 0 1142 storage. Solid state drive (SSD) 1144 contains the operating system and associated files for matching engine 170 .

In the example configuration, three cameras 1112 , 1114 and 1116 are connected to the processing platform (network node) 103 . Each camera has dedicated graphics processing units GPU 1 1162, GPU 2 1164, and GPU 3 1166 for processing images sent by the cameras. It is to be understood that less or more than three cameras may be connected per processing platform. Accordingly, fewer or more GPUs are configured in the network nodes so that each camera has a dedicated GPU for processing image frames received from the cameras. Processor subsystem 1150 , storage subsystem 1130 , and GPUs 1162 , 1164 , and 1166 communicate using bus subsystem 1154 .

Network interface subsystem 1170 is connected to bus subsystem 1154 , forming part of processing platform (network node) 103 . Network interface subsystem 1170 provides interfaces to outside networks, including interfaces to corresponding interface devices in other computer systems. Network interface subsystem 1170 allows processing platforms to communicate over a network by either using cables (or wires) or wirelessly. Wireless radio frequency signals 1175 transmitted by mobile computing device 120 in the region of real space are received by network interface subsystem 1170 (via a wireless access point) for processing by matching engine 170 . Several peripheral devices, such as UI output devices and UI input devices, are also connected to the bus subsystem 1154 forming part of the processing platform (network node) 103 . These subsystems and devices are intentionally not shown in FIG. 11 to improve clarity of description of the present invention. Although the bus subsystem 1154 is schematically shown as a single bus, alternative embodiments of the bus subsystem may use multiple bus bars.

In one embodiment, camera 114 may be implemented using a Chameleon3 1.3 MP Color USB3 Vision (Sony ICX445) with 1288 x 964 resolution, 30 FPS frame rate and 1.3 megapixels per image, with 300 - ∞ The working distance (mm) of the zoom lens has a field of view of 1/3" sensor of 98.2° - 23.8°. Specific implementation

In various embodiments, the above-described system for linking a subject with a user account in a region of real space also includes one or more of the following features.

The system includes a plurality of cameras, a camera in the plurality of cameras generating a sequence of respective images in a corresponding field of view in real space. A processing system is coupled to the plurality of cameras, the processing system includes determining a position of an identified subject represented in the image. The system matches the identified subject to the user account by identifying the location of the mobile device executing the client application in the region of real space, and matches the location of the mobile device to the location of the subject.

In one embodiment, the signal transmitted by the system from the mobile computing device includes an image.

In one embodiment, the signal transmitted by the mobile device includes a radio frequency signal.

In one embodiment, the system includes a set of signature images accessible to the processing system. The processing system includes logic to accept a login communication from a client application on the mobile computing device that identifies the user account prior to matching the user account to a subject identified in the region of real space, and after accepting the login communication, the system Logic to send selected beacon images from a set of beacon images to a client application on a mobile device.

In one embodiment, the processing system sets the status of the selected marker image to assigned. The processing system receives the displayed image of the selected sign image. The processing system identifies the displayed image and matches the identified sign image with a specified sign image from the set of sign images. The processing system matches the location of the mobile device displaying the image of the identified sign located in the region of real space with the identified subject not yet associated. After matching the user account to the identified subject, the processing system sets the status of the identified sign image to available.

In one embodiment, a client application on the mobile computing device transmits accelerometer data to the processing system, and the system uses the accelerometer data transmitted from the mobile computing device to match identified subjects with user accounts.

In one such embodiment, the logic to match an identified subject to a user account includes using accelerometer data transmitted from a plurality of locations over time intervals in a region of real space from a mobile computing device, and using Logic for the derivation of data indicating the location of the identified subject over the time interval in the region of real space.

In one embodiment, the signal transmitted by the mobile computing device includes location data and accelerometer data.

In one embodiment, the signal transmitted by the mobile computing device includes images.

In one embodiment, the signal transmitted by the mobile device includes a radio frequency signal.

A method of associating a subject with a user account in a region of real space is disclosed. It is disclosed that a user account is linked to a client application executable on a mobile computing device. The method includes using a plurality of cameras to generate respective image sequences in corresponding fields of view in real space. Next, the method includes determining a location of the identified subject represented in the image. The method includes matching an identified subject to a user account by identifying a location of a mobile computing device executing a client application in a region of real space. Finally, the method includes matching the location of the mobile computing device to the location of the subject.

In one embodiment, the method also includes setting the state of the selected sign image to Designated, receiving a displayed image of the selected sign image, identifying the displayed sign image, and comparing the identified image to the image from the group. The specified image match for the sign image. The method includes matching the location of a mobile computing device displaying an image of an identified sign located in an area of real space to an identified subject that has not been associated. Finally, the method includes, after matching the user account to the identified subject, setting the status of the identified sign image to available.

In one embodiment, the step of matching the identified subject to the user account further includes using accelerometer data transmitted from the mobile computing device from a plurality of locations over time intervals in a region of real space. The derivation of data indicates the location of the identified subject over time intervals in real space.

In one embodiment, the signal transmitted by the mobile computing device includes location data and accelerometer data.

In one embodiment, the signal transmitted by the mobile computing device includes images.

In one embodiment, the signal transmitted by the mobile device includes a radio frequency signal.

A non-transitory computer readable storage medium having computer program instructions for associating a subject in a region of real space with a user account is revealed. A user account is associated with a client application executable on the mobile computing device, the instructions executing a method when executed on the processor. The method includes using a plurality of cameras to generate respective image sequences in corresponding fields of view in real space. The method includes determining a location of an identified subject represented in the image. The method includes matching an identified subject to a user account by identifying a location of a mobile computing device executing a client application in a region of real space. Finally, the method includes matching the location of the mobile computing device to the location of the subject.

In one embodiment, the non-transitory computer readable storage medium performs a method further comprising the following steps. The method includes setting the state of the selected sign image to assigned, receiving a displayed image of the selected sign image, identifying the displayed sign image, and comparing the identified image to the assigned image from the set of sign images. Image matching. The method includes matching the location of a mobile computing device displaying an image of an identified sign located in an area of real space to an identified subject that has not been associated. After matching the user account to the identified subject, the status of the identified sign image is set to available.

In one embodiment, the non-transitory computer readable storage medium implements the method comprising: by using accelerometer data transmitted from a plurality of locations over time intervals in a region of real space from a mobile computing device, and using The derivation of data indicating the location of the identified subject over the time interval in the region of real space matches the identified subject with a user account.

In one embodiment, the signal transmitted by the mobile computing device includes location data and accelerometer data.

Any data structures and codes described or referenced above are stored according to many implementations in computer readable memory, which includes non-transitory computer readable storage media, which may be capable of storing code for use by a computer system and/or data on any device or media. This includes (but is not limited to) volatile memory, non-volatile memory, Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs), magnetic and optical storage devices (such as disk drives, magnetic tape, CD (Compact Disc), DVD (Digital Versatile Disc or Digital Video Disc)) or other media now known or later developed capable of storing computer-readable media. The previous description was presented to enable completion and use of the disclosed techniques. Various modifications to the disclosed implementations will be apparent, and the general principles defined herein may be applied to other implementations and applications without departing from the spirit and scope of the technology of the present disclosure. Thus, the techniques of the present disclosure are not intended to be limited to the practice shown, but to be accorded the widest scope consistent with the principles and features disclosed herein. The scope of the technology disclosed herein is defined by the appended claims.

100‧‧‧system 102‧‧‧Network Node 103‧‧‧Network Node 110‧‧‧subject tracking engine 112a, 112b, 112n‧‧‧Image recognition engine 114‧‧‧camera 116a, 116b, 116n‧‧‧Aisle 118a, 118b, 118m‧‧‧Mobile computing device 120‧‧‧Mobile Computing Devices 130‧‧‧Training database 140‧‧‧subject database 150‧‧‧user account database 160‧‧‧image database 170‧‧‧Matching Engine 181‧‧‧Internet 202‧‧‧shelves 204‧‧‧shelves 206‧‧‧Camera 208‧‧‧camera 216‧‧‧Vision 218‧‧‧Vision 220‧‧‧Floor 230‧‧‧roof 240‧‧‧subject 250‧‧‧Wireless access points 252‧‧‧Wireless access points 101a, 101b, 101n‧‧‧Network Node 1112‧‧‧Camera 1114‧‧‧camera 1116‧‧‧Camera 1118‧‧‧camera 1122‧‧‧Ethernet connector 1124‧‧‧Ethernet connector 1126‧‧‧Ethernet connector 1128‧‧‧Ethernet connector 1130‧‧‧storage subsystem 1132‧‧‧Master control memory subsystem 1134‧‧‧main random access memory 1136‧‧‧ROM 1140‧‧‧File Storage Subsystem 1142‧‧‧Redundant Array of Independent Disks 1144‧‧‧SSD 1146‧‧‧hard disk drive 1150‧‧‧processor subsystem 1154‧‧‧bus subsystem 1162‧‧‧Graphics Processing Unit 1164‧‧‧Graphics Processing Unit 1166‧‧‧Graphics Processing Unit 1170‧‧‧Network Interface Subsystem 1175‧‧‧radio frequency signal 1181‧‧‧Internet

Figure 1 shows an architectural level schematic of a system in which a matching engine associates a subject identified by a subject tracking engine to a user account associated with a client application executing on a mobile device.

FIG. 2 is a side view showing an aisle in a shopping store with a body of a mobile computing device and a camera arrangement.

3 is a top view showing the aisle of FIG. 2 in a shopping store with a body of a mobile computing device and a camera arrangement.

FIG. 4 illustrates an example data structure for storing joint information of a subject.

FIG. 5 illustrates an example data structure for storing a body including information about associated joints.

6 is a flowchart illustrating process steps for matching identified subjects to user accounts using a badge image displayed on a mobile computing device.

7 is a flow diagram illustrating process steps for using a service location of a mobile computing device to match identified subjects to user accounts.

8 is a flow diagram illustrating process steps for matching identified subjects to user accounts using subject and velocity of the mobile computing device.

9A is a flowchart illustrating process steps for using a network population to match identified subjects to a first portion of user accounts.

9B is a flowchart illustrating process steps for using a network population to match identified subjects to a second portion of user accounts.

9C is a flow diagram illustrating the process steps for using the network population to match identified subjects to user accounts for a third portion of the process.

FIG. 10 is an example architecture in which the four techniques presented in FIGS. 6-9C are applied to regions of real space to reliably match identified subjects to user accounts.

FIG. 11 shows the camera and computer hardware layout used to host the matching engine in FIG. 1 .

100‧‧‧system

101a, 101b, 101n‧‧‧Network node

102‧‧‧Network Node

103‧‧‧Network Node

110‧‧‧subject tracking engine

112a, 112b, 112n‧‧‧Image recognition engine

114‧‧‧camera

116a, 116b, 116n‧‧‧Aisle

118a, 118b, 118m‧‧‧Mobile computing device

120‧‧‧Mobile Computing Devices

130‧‧‧Training database

140‧‧‧subject database

150‧‧‧user account database

160‧‧‧image database

170‧‧‧Matching Engine

181‧‧‧Internet

Claims (24)

A system for associating a subject in a region of real space with a user account associated with a client application executable on a mobile computing device, comprising: a processing system configured to receive data in the A sequence of images of corresponding views in real space, the processing system including logic for determining the location of an identified subject represented in the images, including for executing a client in the region of the real space by identifying The location of the mobile computing device for which the client application on the mobile computing device causes the token The image is displayed on the mobile computing device in the region of the real space, and the logic to match the identified subject to a user account is identified using an image recognition engine that determines a location of the mobile computing device displaying the sign image The location of the mobile computing device. The system of claim 1, wherein the mobile computing device emits a signal usable to indicate the location of the mobile computing device in the region of real space, and the method used to match the identifying subject to a user account Logic uses the emitted signal to identify the location of the mobile computing device. The system of claim 1, wherein the logic for matching the identified subject with a user account is not associated with the user account Personal identification biometric information to operate. The system of claim 1, comprising a set of signage images accessible to the processing system, and wherein the processing system includes a system for matching the user account to an area in the physical space The identifying principal previously received a login communication from the client application on the mobile computing device that identified the user account, and after accepting the login communication, sending a selected one of the set of banner images to the mobile computing device The logic for this client application on . The system of claim 1, wherein the client application on the mobile computing device transmits accelerometer data to the processing system, and the logic for matching the identifying subject to a user account is used from the The accelerometer data transmitted by the mobile computing device. The system of claim 1, wherein the mobile computing device emits a signal usable to indicate the location of the mobile computing device in the region of real space, and said is used to match the identifying entity with a user account The logic includes training a network to identify a location of a mobile computing device in the region of real space based on the signal emitted by the mobile computing device. Such as the system of item 1 of the scope of the patent application, which further includes a log data structure, which includes a series of inventory items for the identification subject, the processing system The system includes associating the log data structure for the matching identified subject to a user account for the identified subject. A system for associating a subject in a region of real space with a user account associated with a client application executable on a mobile computing device, comprising: a processing system configured to receive data in the A sequence of images of corresponding views in real space, the processing system including logic for determining the location of an identified subject represented in the images, including for executing a client in the region of the real space by identifying A location of a mobile computing device for an application, and logic for matching the location of the mobile computing device with the location of the principal to match the identifying principal with a user account, wherein the client application on the mobile computing device transmits location data to The processing system, and the logic for matching the identifier to a user account uses location data communicated from the mobile computing device, and wherein the logic for matching the identifier to a user account comprises using the logic for sending location data from the mobile computing device from multiple locations over a time interval in the region of real space, the logic further includes: all other mobile computing devices that decide to send location information that does not match a user account the device is separated from the mobile computing device by a predetermined distance; determining the closest unmatched identifying subject to the mobile computing device; and The unmatched identity is matched to a user account of the client application executing on the mobile computing device. A method of associating a subject in a region of real space with a user account associated with a client application executable on a mobile computing device, the method comprising: receiving a corresponding view in the real space a sequence of images of a plurality of images; determine the location of an identified subject represented in the sequence of images; associate the identified subject with a user account by identifying the location of a mobile computing device executing a client application in the region of real space matching; and matching the location of the mobile computing device with the location of the subject, wherein the client application on the mobile computing device causes a sign image to be displayed on the mobile computing device in the region of the real space, and The step of matching the identified subject to the user account further includes identifying the location of the mobile computing device using an image recognition engine that determines the location of the mobile computing device displaying the sign image. The method of claim 9, wherein: the mobile computing device emits a signal usable to indicate the location of the mobile computing device in the region of real space; and the matching of the identifying subject to a user account The steps further include using the transmitted signal to identify a location of the mobile computing device. The method of claim 9, wherein the matching of the identifying subject with the user account is performed without using personally identifiable biometric information associated with the user account. The method of claim 9, further comprising: receiving a login communication from a client application on a mobile computing device that identifies the user account before matching the user account to an identified subject in the region of the physical space , and after accepting the login communication, sending a selected sign image from the set of sign images to the client application on the mobile computing device. The method of claim 9, wherein the step of matching the identification subject with the user account further includes: using accelerometer data transmitted from the mobile computing device. The method of claim 9, wherein the step of matching the identified subject with the user account further includes: training the network to identify mobile computing in the area of the real space based on the signal emitted by the mobile computing device The location of the device. The method of claim 9 further comprising associating a log data structure comprising a series of inventory items for the matching identified subject to a user account for the identified subject. A method of associating a subject in a region of real space with a user account associated with a client application executable on a mobile computing device, the method comprising: receiving a corresponding view in the real space a sequence of images of a plurality of images; determine the location of an identified subject represented in the sequence of images; associate the identified subject with a user account by identifying the location of a mobile computing device executing a client application in the region of real space matching; and matching the location of the mobile computing device to the location of the subject, wherein the step of matching the identified subject to a user account further includes: using location data transmitted from the mobile computing device, and wherein the method further including: matching the identifying subject to a user account using the location data of the mobile computing device transmitted from a plurality of locations over time intervals in the region of real space from the mobile computing device, including: determining to transmit unmatched usage all other mobile computing devices that are separated by a predetermined distance from the mobile computing device; determine the closest unmatched identifying subject to the mobile computing device; and associate the unmatched identifying subject on the mobile computing device The user account matches the executed client application. A non-transitory computer-readable storage medium imprinted with computer program instructions for connecting subjects in regions of real space to Linking a user account, the user account is linked to a client application executable on a mobile computing device, and when executed on a processor, the instructions execute a method comprising: receiving a corresponding field of view in the real space a sequence of images; determining a location of an identified subject represented in the sequence of images; matching the identified subject to a user account by identifying a location of a mobile computing device executing a client application in the region of real space and matching the location of the mobile computing device with the location of the subject, wherein the client application on the mobile computing device causes a sign image to be displayed on the mobile computing device in the region of the real space, and the Matching the identifying subject to the user account further includes identifying the location of the mobile computing device using an image recognition engine that determines the location of the mobile computing device displaying the sign image. The non-transitory computer-readable storage medium of claim 17, wherein the mobile computing device emits a signal usable to indicate the location of the mobile computing device in the region of real space, and said identifying subject The step of matching the user account further includes using the transmitted signal to identify the location of the mobile computing device. The non-transitory computer-readable storage medium according to claim 17 of the patent application, wherein the step of matching the identification subject with the user account is performed without using personal identification biometric information associated with the user account. According to the non-transitory computer-readable storage medium of claim 17, said implementing the method further includes: before matching the user account to identify the subject in the region of the real space, from the identification of the user account A client application on the mobile computing device receives the login communication, and after accepting the login communication, sends a selected sign image from the set of sign images to the client application on the mobile computing device. The non-transitory computer-readable storage medium of claim 17, wherein the step of matching the identification subject with a user account further includes: using accelerometer data transmitted from the mobile computing device. The non-transitory computer-readable storage medium of claim 17, wherein the matching of the identified subject with the user account further includes: training the network to identify the real space based on the signal emitted by the mobile computing device The location of the mobile computing device in the area. The non-transitory computer-readable storage medium of claim 17 further includes associating a log data structure including a series of inventory items for the matching identified subject to a user account for the identified subject. A non-transitory computer-readable storage medium imprinted with computer program instructions for connecting subjects in regions of real space to Linking a user account, the user account is linked to a client application executable on a mobile computing device, and when executed on a processor, the instructions execute a method comprising: receiving a corresponding field of view in the real space a sequence of images; determining a location of an identified subject represented in the sequence of images; matching the identified subject to a user account by identifying a location of a mobile computing device executing a client application in the region of real space and matching the location of the mobile computing device to the location of the subject, wherein said step of matching the identified subject to a user account further comprises: using location data transmitted from the mobile computing device, and wherein said performing the The method further includes matching the identifying subject to a user account using the location data of the mobile computing device transmitted from a plurality of locations over a time interval in the region of real space, including: determining to transmit an unidentified all other mobile computing devices matching the location information of the user account are separated from the mobile computing device by a predetermined distance; determining the closest unmatched identifying subject to the mobile computing device; The user account of the client application executing on the device matches.

Images