JP7228671B2 - Store Realog Based on Deep Learning - Google Patents

Description

priority application

No. 62/703,785 (Attorney Docket No. STCG 1006-1), filed July 26, 2018, which is incorporated herein by reference, and January 24, 2019 The benefit of filing US patent application Ser. No. 16/256,355 (Attorney Docket No. STCG 1007-1) is claimed. Said U.S. patent application Ser. In a continuation-in-part of U.S. patent application Ser. U.S. Patent Application Serial No. 15/907,112 (Attorney Docket No. STCG 1002-1) filed February 27, 2018 (now U.S. Patent No. 10,133,933, issued November 20, 2018) No. 15/945,473, filed April 4, 2018 (Attorney Docket No. STCG 1005-1), which is a continuation-in-part of US Patent Application No. These US patent applications are incorporated herein by reference.

The present invention relates to a system for tracking inventory items within areas of real space containing inventory display structures.

Determining the quantity and location of various inventory items stocked in an inventory display structure within an area of a real space such as a shopping store is required for efficient operation of the shopping store. A subject within an area of real space, such as a customer, picks an item from a shelf and places the item in their shopping cart or basket. Also, if the customer does not want to purchase the item, the customer can put the item back on the same shelf or another shelf. Thus, over a period of time, inventory items may be removed from designated locations on the shelf and distributed to other shelves within the shopping store. In some systems, stocked item quantities are available after significant delays due to the need to concatenate receipts and stock inventory. Delays in the availability of information about the quantity of products stocked in a shopping store can affect customer purchasing decisions and the actions of store managers to order more of high-demand inventory. be.

It would be desirable to provide a system that can more effectively and automatically provide the quantity of items stocked on a shelf in real time and identify the location of items on the shelf.

A system for tracking inventory in an area of real space and a method of operating the system are provided. Multiple cameras or other sensors generate respective image sequences of corresponding fields of view in real space. The system includes processing logic coupled to a plurality of sensors and using image sequences generated by at least two sensors to identify inventory events. The system tracks inventory items within areas of real space in response to inventory events.

Systems and methods are provided for tracking inventory items within areas of real space. Multiple cameras or other sensors generate respective image sequences of corresponding fields of view in real space. The field of view of each sensor overlaps the field of view of at least one other sensor in the plurality of sensors. The system uses a sequence of images produced by at least two sensors in the plurality of sensors to identify inventory events. The system tracks the location of inventory items within areas of real space in response to inventory events.

In one embodiment, an inventory event includes an item identifier, a put or take indicator, a position represented by a position along three axes of an area in real space, and a timestamp. The system may include or have access to a memory storing a data set defining a plurality of cells having coordinates within an area of real space. The system includes logic that matches inventory item locations with cell coordinates, and maintains data representing inventory items that match cells within a plurality of cells. An area of real space can include multiple inventory locations. Coordinates of cells within the plurality of cells can be correlated with inventory locations or portions of inventory locations within the plurality of inventory locations. The system includes logic that uses each count of inventory events to calculate scores during scoring for inventory items that have a matching position in a particular cell. The logic to calculate a cell's score uses the total put and take inventory weighted by the separation between the put and take time stamps and the time of scoring. The system includes logic to store the scores in memory.

In one embodiment, a system includes logic to render a display image representing a cell and the score of the cell within a plurality of cells. In this embodiment, the score is represented by a color change in the displayed image representing the cell. The system includes logic that selects a set of inventory items for each cell based on the scores. In one embodiment, an area of real space includes a plurality of inventory locations, and coordinates of cells within the plurality of cells correlate to inventory locations within the plurality of inventory locations. In this embodiment, a data set in memory defines a plurality of cells having coordinates within an area of real space.

The system may include or have access to a memory that stores a planogram identifying inventory locations and inventory items located at the inventory locations within an area of real space. The planogram may also include information regarding the portion of inventory locations designated for particular inventory items. A planogram can be generated based on a plan for placement of inventory items on inventory locations within an area of real space.

The system includes logic to maintain data representing inventory items matching cells in a plurality of cells. The system may also include logic to determine misplaced items by comparing data representing inventory items that match the cell to the planogram.

The system identifies the location of an inventory item within an area of real space based on the accumulation of data about the item and its location in detected inventory events, as discussed herein, referred to herein as a "rear A data structure called a "program" can be generated and stored in memory. The data in the realogogram is compared to the data in the planogram to determine how inventory items are placed in the area compared to the plan, such as locating misplaced items be able to. Realog may also be used to locate inventory items within three-dimensional cells and correlate those cells with inventory locations within a store, as may be determined, for example, from a planogram or other map of inventory locations. can be processed. Also, re-logograms can be processed to track activity associated with a particular inventory item at various locations within an area. Other uses of realograms are also possible.

A system and method are provided for tracking inventory items within an area of real space that includes an inventory display structure. The system includes multiple cameras positioned above an inventory display structure. A camera produces a respective image sequence of an inventory display structure within a corresponding field of view in real space. The field of view of each camera overlaps the field of view of at least one other camera in the plurality of cameras. A dataset defines a plurality of cells having coordinates within an area of real space. Datasets are stored in memory. The system processes image sequences generated by multiple cameras to locate inventory events in three dimensions within an area of real space. In response to the inventory event, the system includes logic to determine the closest cell in the dataset based on the location of the inventory event. The system includes logic that, at scoring, computes scores for inventory items associated with inventory events that have locations that match a particular cell using the respective count of inventory events.

In one embodiment, the system includes logic to select a set of inventory items for each cell based on the score. In one embodiment, an inventory event includes an item identifier, a put or take indicator, a position represented by a position along three axes of an area in real space, and a timestamp. In one embodiment, a system includes a dataset defining a plurality of cells represented as a two-dimensional grid having coordinates within an area of real space. A cell can be correlated with a front view portion of an inventory location. The processing system includes logic to determine the closest cell based on the location of the inventory event. In one embodiment, the system includes a dataset defining a plurality of cells represented as a three-dimensional grid having coordinates within an area of real space. A cell can be correlated with a portion of volume on an inventory location. The processing system includes logic to determine the closest cell based on the location of the inventory event. A put indicator identifies that the item has been placed in the inventory location, and a take indicator identifies that the item has been removed from the inventory location.

In one embodiment, the logic for processing image sequences generated by multiple cameras comprises an image recognition engine. An image recognition engine generates a dataset representing elements in the image that correspond to the hand. The system includes logic to perform analysis of data sets from image sequences from at least two cameras to determine the location of inventory events in three dimensions. The image recognition engine comprises a convolutional neural network.

In one embodiment, the system has logic that calculates a cell's score using the sum of pick and put inventory items weighted by the separation between the put and pick time stamps and the time of scoring. including. The score is stored in memory. In one embodiment, the logic for determining the closest cell in the dataset based on the location of the inventory event comprises: calculating the distance from the location of the inventory event to the cell in the dataset; and matching inventory events with cells using the method.

Methods and computer program products that can be executed by computer systems are also described herein.

The functionality described herein identifies inventory events that include items associated with the inventory event, links them to cells within a plurality of cells that have coordinates within an area of real space, and stores a store realog. relating to, for example, the type of image data to be processed, what processing of the image data to perform, and how to reliably determine operations from the image data, including but not limited to updating , presents a complex problem in computer engineering.

Other aspects and advantages of the invention can be understood from a review of the following drawings, detailed description, and claims.

1 shows an architecture-level schematic diagram of a system in which a store inventory engine and a store re-alog engine track inventory items within an area of a real space containing an inventory display structure; FIG.

1 is a side view of an aisle within a shopping store showing subjects within the shopping store, inventory display structure, and camera placement; FIG.

2B is a perspective view of the aisle inventory structure of FIG. 2A showing a subject removing items from a shelf in the inventory structure; FIG.

Fig. 3 shows an example of 2D and 3D maps of shelves in an inventory display structure;

4 illustrates an exemplary data structure for storing joint information of a subject;

4 shows an exemplary data structure for storing an object including associated joint information.

2B is a top view of an inventory display structure of the in-aisle shelf units of FIG. 2A in a shopping store showing selection of a shelf within the inventory display structure based on the location of an inventory event indicating an item being picked from the shelf; FIG.

FIG. 10 illustrates an example of a log data structure that can be used to store a subject's shopping cart or inventory items stocked on a shelf or in a shopping store; FIG.

FIG. 4 is a flow chart showing the process steps for determining inventory on shelves and in a shopping store based on placement and pick-up locations of inventory.

9 is an exemplary architecture in which the techniques illustrated in the flowchart of FIG. 8 can be used to determine inventory on shelves within an area of real space.

9 is an exemplary architecture in which the techniques illustrated in the flowchart of FIG. 8 can be used to update a store inventory data structure;

FIG. 11 illustrates the discretization of shelves in portions within an inventory display structure using a two-dimensional (2D) grid; FIG.

Dispersed inventory from a designated position on a portion of a shelf within an inventory display structure to other positions on the same shelf and after one day to different shelf positions within other inventory structures within the shopping store. 1 is an illustration of a rearogram using a three-dimensional (3D) grid of shelves showing the location of items;

11B is an example showing the realogram of FIG. 11A displayed on a user interface of a computing device; FIG.

Fig. 3 is a flow chart showing the processing steps for calculating a realog of inventory items stocked on shelves of an inventory display structure within a shopping store based on placement and pick-up locations of the inventory items;

FIG. 10 is a flow chart showing processing steps for determining restocking of inventory items using a rearogram; FIG.

4 is an exemplary user interface displaying restock notifications for inventory items.

FIG. 4 is a flow chart showing the process steps for determining planogram compliance using a realogram; FIG.

4 is an exemplary user interface displaying misplaced item notifications for inventory items.

FIG. 4 is a flow chart showing the process steps for adjusting the confidence score probabilities of inventory forecasts using a realog.

2 is a camera and computer hardware configuration configured to host the inventory consolidation engine and store re-log engine of FIG. 1;

The following description is presented to enable any person skilled in the art to make and use the invention, and is provided in the context of particular applications and requirements thereof. Various modifications to the disclosed embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be adapted to other embodiments and applications without departing from the spirit and scope of the invention. can be applied to Accordingly, the invention is not intended to be limited to the embodiments shown, but is to be accorded the widest scope consistent with the principles and features disclosed herein.
[System Overview]

A system and various implementations of the subject technology are described with reference to FIGS. The system and processing are described with reference to FIG. 1, which is an architectural level schematic diagram of the system according to the present embodiment. Since FIG. 1 is an architectural diagram, certain details have been omitted to improve the clarity of the description.

The description of FIG. 1 is organized as follows. First, the elements of the system are described, then their interconnections. The use of the elements in the system will now be described in more detail.

FIG. 1 provides a block diagram level illustration of system 100 . The system 100 includes a camera 114, image recognition engines 112a, 112b and 112n hosted by network nodes, a store inventory engine 180 located in a network node (or nodes) 104 on the network, a network node on the network. A store realog engine 190 located within a node (or nodes) 104, a network node 102 hosting an object tracking engine, a map database 140, an inventory event database 150, a planogram and inventory database 160, a realog • includes a database 170 and one or more communication networks 181; A network node can host only one image recognition engine, or multiple image recognition engines as described herein. The system can also include subject databases and other supporting data.

As used herein, a network node is an addressable node that is connected to a network and capable of sending, receiving, or transferring information to and from other network nodes via communication channels. A hardware device or a virtual device. Examples of electronic devices that can be deployed as hardware network nodes include computers of all kinds, workstations, laptop computers, handheld computers, and smart phones. A network node may be implemented in a cloud-based server system. Multiple virtual devices configured as network nodes can be implemented using a single physical device.

For clarity, only three network nodes hosting image recognition engines are shown in system 100 . However, any number of network nodes hosting image recognition engines can be connected to object tracking engine 110 via network 181 . Similarly, the image recognition engine, object tracking engine, store inventory engine, store realog engine, and other processing engines described herein run using multiple network nodes in a distributed architecture. be able to.

The interconnection of the elements of system 100 will now be described. Network 181 includes network nodes 101a, 101b, and 101n, which host image recognition engines 112a, 112b, and 112n, respectively; network node 104, which hosts store inventory engine 180; It couples node 106 , network node 102 hosting tracking engine 110 , map database 140 , inventory event database 150 , inventory database 160 and realogram database 170 . Camera 114 is connected to object tracking engine 110 via network nodes that host image recognition engines 112a, 112b, and 112n. In one embodiment, cameras 114 are installed in a shopping store (such as a supermarket), and a set (two or more) of cameras 114 with overlapping fields of view are placed over each aisle to capture images of the real space within the store. do. In FIG. 1, two cameras are positioned above aisle 116a, two cameras are positioned above aisle 116b, and three cameras are positioned above aisle 116n. Cameras 114 are placed on the passageway with overlapping fields of view. In such embodiments, the cameras are configured with the goal that a customer moving through the aisles of the shopping store is within the field of view of more than one camera at any given time.

The cameras 114 can be synchronized in time with each other so that images are acquired at the same time or close in time and at the same image capture rate. Cameras 114 may send respective continuous image streams at a predetermined rate to network nodes hosting image recognition engines 112a-112n. Images acquired at all cameras covering an area of real space at the same time or nearby in time are viewed by the processing engine as synchronized images representing different views of an object having a fixed position in real space. Synchronous in the sense that they can be identified. For example, in one embodiment, cameras transmit image frames at a rate of 30 frames per second (fps) to respective network nodes hosting image recognition engines 112a-112n. Along with the image data, each frame has a time stamp, camera identification information (abbreviated as “camera ID”), and frame identification information (abbreviated as “frame ID”). Other embodiments of the disclosed technology use various types of sensors, such as infrared image sensors, radio frequency image sensors, ultrasonic sensors, thermal sensors, lidar, etc., to generate this data. be able to. Multiple types of sensors may be used in addition to camera 114 that produces RGB color output, including, for example, infrared or high frequency image sensors. The multiple sensors are time-synchronized with each other so that frames are acquired by the sensors at the same time or close in time and at the same frame capture rate. In all of the embodiments disclosed herein, sensors other than cameras, or multiple types of sensors, may be used to generate the image sequences used.

Cameras installed on the corridor are connected to their respective image recognition engines. For example, in FIG. 1, two cameras located on corridor 116a are connected to network node 101a, which hosts image recognition engine 112a. Similarly, two cameras placed on the corridor 116b are connected to the network node 101b that hosts the image recognition engine 112b. Each image recognition engine 112a-112n hosted within a network node 101a-101n separately processes image frames received from a respective camera in the illustrated example.

In one embodiment, each image recognition engine 112a, 112b, and 112n is implemented as a deep learning algorithm, such as a convolutional neural network (CNN for short). In such embodiments, the CNN is trained using training database 150 . In the embodiments described herein, image recognition of objects in real space is based on identifying and grouping recognizable joints in the image, and joint groups are attributed to individual objects. be able to. For this joint-based analysis, the training database 150 has collected a large number of images for each different type of joint for the subject. In an exemplary embodiment of a shopping store, the subject is a customer moving through the aisles between the shelves. In an exemplary embodiment, during CNN training, system 100 is referred to as a "training system." After training the CNN using the training database 150, the CNN is switched to production mode to process images of customers in the shopping store in real time.

In an exemplary embodiment, during production, system 100 is referred to as a runtime system (also referred to as an inference system). Each image recognizer's CNN generates an array of joint data structures for the images in each image stream. In the embodiments described herein, an array of joint data structures is generated for each processed image such that each image recognition engine 112a-112n generates an output stream of arrays of joint data structures. Generate. These arrays of joint data structures from cameras with overlapping fields of view are further processed to form joint groups and identify such joint groups as objects. The system can use the identifier "subject ID" to identify and track the subject while the subject resides within an area of real space.

The subject tracking engine 110 is hosted on the network node 102 and, in this example, receives a continuous stream of arrays of subject joint data structures from the image recognition engines 112a-112n. The object tracking engine 110 processes the array of joint data structures and transforms the coordinates of the elements in the array of joint data structures corresponding to the various sequences of images into candidate joints having coordinates in real space. For each set of synchronous images, the combination of candidate joints identified over real space can be thought of as resembling a galaxy of candidate joints for analogy purposes. At each subsequent time point, motion of the candidate joint is recorded as the galaxy changes over time. Object tracking engine 110 identifies objects within an area of real space at a point in time.

The tracking engine 110 uses logic to identify groups or sets of candidate joints having coordinates in real space as objects in real space. For analogy purposes, each set of candidate points resembles a constellation of candidate joints at each instant. The constellation of candidate joints can move over time. A time-series analysis of the output of the object tracking engine 110 over time identifies the movement of objects within areas of real space.

In an exemplary embodiment, the logic for identifying the set of candidate joints includes a heuristic function based on physical relationships between the subject's joints in real space. These heuristic functions are used to identify a set of candidate joints as objects. The set of candidate joints includes individual candidate joints that have heuristic parameter-based relationships with other individual candidate joints, and individual subjects within a given set that have been or can be identified as individual subjects. Contains a subset of candidate joints.

In the example of a shopping store, customers (also referred to as subjects above) move through aisles and open spaces. A customer picks up an item from an inventory position on a shelf within an inventory display structure. In one example of an inventory display structure, shelves are arranged at various levels (or heights) from the floor and inventory items are stocked on the shelves. Shelves may be fixed to walls or arranged as free-standing shelves forming an aisle within a shopping store. Other examples of inventory display structures include pegboard shelving, magazine shelving, carousel shelving, warehouse shelving, and refrigerated shelving units. Inventory may also be stocked in other types of inventory display structures such as stacked wire baskets, dump bins, and the like. Customers can also return items to the same shelf from which they were removed or to a different shelf.

The system includes a store inventory engine 180 (hosted on network node 104) to update inventory at inventory locations within the shopping store as customers place and remove items from the shelves. . The store inventory engine updates the inventory data structure of the inventory location by indicating the identifiers (such as inventory keeping units or SKUs) of the inventory items placed in the inventory location. The inventory consolidation engine also updates the shopping store's inventory data structure by updating those quantities stocked in the shopping store. Inventory locations and store inventory data are stored in inventory database 160 along with customer inventory data (also referred to as an inventory log data structure or shopping cart data structure).

The store inventory engine 180 provides the status of inventory items in inventory locations. However, it is difficult to determine which inventory items are located on which portion of the shelf at any given time. This is important information for shopping store managers and employees. Inventory items can be placed in inventory locations based on a planogram that identifies shelves and locations on the shelves where the inventory items are planned to be stocked. For example, ketchup bottles may be stocked in predetermined left portions of all shelves in an inventory display structure forming a row arrangement. Over time, customers remove ketchup bottles from the shelves and place them in their respective baskets or shopping carts. Some customers may return the ketchup bottle to another portion of the same shelf within the same inventory display structure. Customers may also return ketchup bottles to shelves in other inventory structures within the shopping store. Store realog engine 190 (hosted on network node 106) generates a realog that can be used to identify the portion of the shelf where the ketchup bottle is placed at time "t". This information can be used by the system to generate a notification to the employee who has the location of the misplaced ketchup bottle.

This information is also used across inventory items within areas of real space to generate a data structure, referred to herein as a realog, that keeps track of the location of inventory items within areas of real space. obtain. A representation of the shopping store generated by the store rearogram engine 190 that reflects the current state of the inventory and, in some embodiments, the state of the inventory at a specified time "t" over an interval of time. Realogograms may be stored in realog database 170 .

The actual communication path to network node 104 hosting store inventory engine 170 and network node 106 hosting store realog engine 190 via network 181 may be over public and/or private networks. can be point-to-point. Communication can occur over various networks 181, such as private networks, VPNs, MPLS circuits, or the Internet, using suitable application programming interfaces (APIs) and data exchange formats, such as REST (Representational State Transfer). ), JSON (JavaScript™ Object Notation), XML (Extensible Markup Language), SOAP (Simple Object Access Protocol), JMS (Java™ Message Service), and/or the Java Platform Module System, etc. be able to. All communications can be encrypted. Communications are generally carried out over LANs (Local Area Networks), WANs (Wide Area Networks), telephone networks (Public Switched Telephone Networks) via protocols such as EDGE, 3G, 4G LTE, Wi-Fi, and WiMAX. PSTN)), Session Initiation Protocol (SIP), wireless networks, point-to-point networks, star networks, token ring networks, hub networks, the Internet (including mobile Internet), and the like. Additionally, various authorization and authentication techniques such as username/password, open authorization (OAuth), Kerberos, SecureID, digital certificates, etc. can be used to secure communications.

The technology disclosed herein can be used in database systems, multi-tenant environments, or database implementations compatible with Oracle™, relational database implementations compatible with IBM DB2 Enterprise Server™. , MySQL(TM) or PostgreSQL(TM) compatible relational database implementations or Microsoft SQL Server(TM) compatible relational database implementations, or Vampire(TM) ), a non-relational database implementation compatible with Apache Cassandra™, a non-relational database implementation compatible with BigTable™, or HBase™ ) or any non-relational database implementation of NoSQL™, such as a DynamoDB™ compatible non-relational database implementation. Further, the disclosed technology can be used with various programming models such as MapReduce™, bulk synchronous programming, MPI primitives, or Apache Storm™, Apache Spark™, Apache Kafka™, Apache Flink™ ), Truviso™, Amazon Elasticsearch Service™, Amazon Web Services (AWS)™, IBM Info-Sphere™, Borealis™, and Yahoo! It can be implemented using various scalable batch and stream management systems such as S4™.

[Camera layout]

Camera 114 is positioned to track an articulated object (or entity) in three-dimensional (abbreviated 3D) real space. In an exemplary embodiment of a shopping store, the physical space may include an area of the shopping store where items for sale are stacked on shelves. A point in real space can be represented by an (x, y, z) coordinate system. Each point in the area of real space to which the system is applied is covered by the fields of view of two or more cameras 114 .

In a shopping store, shelves and other inventory display structures can be arranged in a variety of ways, such as along the sidewalls of the shopping store, or in rows forming aisles, or in combinations of the two configurations. FIG. 2A shows the arrangement of shelving unit A 202 and shelving unit B 204 forming aisle 116a, viewed from one end of aisle 116a. Two cameras, camera A 206 and camera B 208, are positioned in the aisle 116a at a predetermined distance from the shopping store ceiling 230 and floor 220 above shelf-like inventory display structures such as shelf unit A 202 and shelf unit B 204. placed above. Camera 114 comprises a camera positioned thereon having a field of view encompassing respective portions of the inventory structure and floor area in real space. As shown in FIG. 2A, the field of view 216 of camera A 206 and the field of view 218 of camera B 208 overlap each other. The coordinates in real space of the members of the set of candidate joints identified as the subject identify the location of the subject within the floor area.

In an exemplary embodiment of a shopping store, the real space may include all floors 220 within the shopping store. Cameras 114 are positioned and oriented such that the floor 220 and shelf areas are viewed by at least two cameras. Camera 114 also covers the floor space in front of shelves 202 and 204 . The angles of the cameras are selected to have both steep, straight and angled points of view, which gives a more complete body image of the customer. In one embodiment, cameras 114 are configured at eight feet or higher throughout the shopping store. FIG. 13 shows an explanatory diagram of such an embodiment.

In FIG. 2A, subject 240 is standing beside shelving unit B 204 of an inventory display structure, with one hand positioned near a shelf (not visible) within shelving unit B 204 . FIG. 2B is a perspective view of shelving unit B 204 with four shelves, shelf 1, shelf 2, shelf 3, and shelf 4, located at different heights from the floor. Inventory items are stocked on these shelves.

[3D scene generation]

A position in real space is represented as a (x, y, z) point in the real space coordinate system. 'x' and 'y' represent locations on a two-dimensional (2D) plane that may be a floor 220 of a shopping store, and the value 'z' is a point on the 2D plane on the floor 220 in one configuration. Height. The system combines 2D images from two or more cameras to generate 3D locations of joints and inventory events (shelf and pick off) within areas of real space. This section describes the process for generating 3D coordinates for joints and inventory events. The process is also called 3D scene generation.

Before using the system 100 in training or inference mode to track inventory, two types of camera calibration are performed: internal and external. In internal calibration, the internal parameters of camera 114 are calibrated. Examples of internal camera parameters include focal length, principal point, skew, fisheye factor, and the like. Various techniques for internal camera calibration can be used. One such technique is presented by Zhang in "Flexible New Approaches for Camera Calibration," published in IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 22, No. 11, November 2000. there is

In external calibration, the extrinsic camera parameters are calibrated to generate mapping parameters for transforming 2D image data into 3D coordinates in real space. In one embodiment, one articulated subject, such as a person, is introduced into real space. The articulated subject moves in real space on a path passing through the field of view of each camera 114 . At any given point in real space, the articulated object is within the field of view of at least two cameras forming a 3D scene. However, the two cameras have different views of the same 3D scene in their respective two-dimensional (2D) image planes. A feature in the 3D scene, such as the left wrist of an articulated subject, is viewed by two cameras at different positions in their respective 2D image planes.

Point correspondences are established between all camera pairs that have overlapping fields of view for a given scene. Since each camera has a different field of view of the same 3D scene, the point correspondence is two pixel positions (one position from each camera with overlapping field of view) representing the projection of the same point in the 3D scene. For external calibration, a number of point correspondences are identified for each 3D scene using the results of the image recognition engines 112a-112n. The image recognition engine identifies joint positions as (x,y) coordinates, eg, row and column numbers, of pixels in the 2D image plane of each camera 114 . In one embodiment, the joint is one of nineteen different types of joints of the articulated subject. As the articulated object is moved through the fields of view of the different cameras, the tracking engine 110 captures the (x,y) coordinates of each of the 19 different types of joints of the articulated object used for calibration from the cameras 114 for each image. receive from

For example, consider the case where an image from camera A and an image from camera B were both captured at the same point in time with overlapping fields of view. There are pixels in the image from camera A that correspond to pixels in the synchronized image from camera B, and there are particular points on some object or surface within the fields of view of both camera A and camera B that are both image frame pixels. For external camera calibration, a number of such points are identified and called corresponding points. Since there is one articulated object in the field of view of camera A and camera B during calibration, the main joint of this articulated object, eg the center of the left wrist, is identified. If these major joints are visible in the image frames from both camera A and camera B, they are assumed to represent corresponding points. This process is repeated for many image frames to build a large set of corresponding points for all camera pairs with overlapping fields of view. In one embodiment, images are streamed from all cameras at a rate of 30 FPS (frames per second) or higher at a resolution of 720 pixels in full RGB (red, green, and blue) color. These images are in the form of one-dimensional arrays (also called flat arrays).

The multiple images collected above for an articulated subject can be used to determine corresponding points between cameras with overlapping fields of view. Consider two cameras A and B with overlapping fields of view. A plane passing through the camera centers of cameras A and B and the joint positions (also called feature points) of the 3D scene is called an "epipolar plane", and the intersection of the epipolar plane and the 2D image planes of cameras A and B is an "epipolar line". defined as Given these correspondence points, the correspondence points from camera A can be accurately mapped to epipolar lines in camera B's field of view that are guaranteed to intersect the corresponding points in camera B's image frame. A transform is determined. A transform is generated using the image frames collected above for the articulated subject. This transformation is known in the art to be non-linear. Furthermore, in general form, it is known that radial distortion correction of each camera lens is required, as well as non-linear coordinate transformations moving into and out of the projected space. In external camera calibration, an approximation to the ideal nonlinear transform is determined by solving a nonlinear optimization problem. This nonlinear optimization function is performed by the object tracking engine 110 to identify the same joints in the output (array of joint data structures) of the various image recognition engines 112a-112n processing images of the camera 114 with overlapping fields of view. used. The results of internal camera calibration and external camera calibration are stored in calibration database 170 .

Various techniques can be used to determine the relative positions of points in the image of camera 114 in real space. For example, Longuet-Higgins has published "A computer algorithm for reconstructing a scene from two projects" (Nature, Vol. 293, September 10, 1981). In this paper, it is presented to compute the 3D structure of a scene from correlated pairs of perspective projections when the spatial relationship between the two projections is unknown. The Longuet-Higgins paper presents a technique for determining the position of each camera relative to other cameras in real space. In addition, the technique allows triangulation of articulated objects in real space, using images from cameras 114 with overlapping fields of view to identify z-coordinate values (height above floor). An arbitrary point in the real space, for example, the end of a corner shelf unit in the real space, is defined as the (0, 0, 0) point on the (x, y, z) coordinate system in the real space.

In one embodiment of the present technology, parameters for external calibration are stored in two data structures. A first data structure stores intrinsic parameters. The intrinsic parameters represent the projective transformation from 3D coordinates to 2D image coordinates. The first data structure contains specific parameters for each camera, as shown below. All data values are floating point numbers. This data structure stores a 3x3 eigenmatrix denoted as 'K' and the distortion coefficients. The distortion coefficients include 6 radial distortion coefficients and 2 tangential distortion coefficients. Radial distortion occurs when a ray of light bends more near the edge of the lens than at its optical center. Tangential distortion occurs when the lens and image plane are not parallel. The data structure below shows the values for the first camera only. Similar data is stored for all cameras 114 .

{
1: {
K: [[x, x, x], [x, x, x], [x, x, x]],
distortion_coefficients: [x, x, x, x, x, x, x, x]
},
......
}

The second data structure contains, for each camera pair, a 3x3 fundamental matrix (F), a 3x3 essential matrix (E), a 3x4 projection matrix (P), a 3x3 rotation matrix (R), and Store the 3×1 translation vector (t). This data is used to transform points in one camera's frame of reference to another camera's frame of reference. Eight homography coefficients are also stored for each pair of cameras to map the plane of the floor 220 from one camera to another. A fundamental matrix is a relationship between two images of the same scene that constrains where projections of points from the scene can occur in both images. The essential matrix is also the relationship between two images of the same scene with the cameras calibrated. A projection matrix gives a vector space projection from the 3D real space to a subspace. Rotation matrices are used to perform rotations in Euclidean space. A translation vector 't' represents a geometric transformation that moves all points in a figure or space the same distance in a given direction. The homography floor coefficients are used to combine images of object features on the floor 220 viewed by cameras with overlapping fields of view. A second data structure is shown below. Similar data is stored for all camera pairs. As mentioned above, x represents a floating point value.

{
1: {
2: {
F: [[x, x, x], [x, x, x], [x, x, x]],
E: [[x, x, x], [x, x, x], [x, x, x]],
P: [[x, x, x, x], [x, x, x, x], [x, x, x, x]],
R: [[x, x, x], [x, x, x], [x, x, x]],
t: [x, x, x],
homography_floor_coefficients: [x, x, x, x, x, x, x, x]
}
},
.......
}

[Two-dimensional map and three-dimensional map]

An inventory location, such as a shelf within a shopping store, may be identified by a unique identifier (eg, shelf ID). Similarly, a shopping store may be identified by a unique identifier (eg, store ID). Two-dimensional (2D) and three-dimensional (3D) map databases 140 identify inventory locations within areas of real space along their respective coordinates. For example, in a 2D map, locations within the map define a two-dimensional area on a plane formed perpendicular to the floor 220, the XZ plane, as shown in FIG. The map defines areas of inventory locations in which inventory items are located. In FIG. 3, a 2D view 360 of shelf 1 in shelf unit B 204 is formed by four coordinate locations (x1,z1), (x1,z2), (x2,z2), and (x2,z1). Denotes an area and defines a 2D region in which inventory items are placed on the shelf 1 . A similar 2D region is defined for every inventory location within every shelf unit (or other inventory display structure) within the shopping store. This information is stored in map database 140 .

In a 3D map, locations in the map define three-dimensional regions in real 3D space defined by X, Y, and Z coordinates. The map defines the inventory location volume in which inventory items are to be placed. In FIG. 3, a 3D view 350 of shelf 1 in shelf unit B 204 has eight coordinate locations (x1, y1, z1), (x1, y1, z2), (x1, y2, z1) defining a 3D region. , (x1, y2, z2), (x2, y1, z1), (x2, y1, z2), (x2, y2, z1), (x2, y2, z2); is placed within that 3D region on the shelf 1 . Similar 3D regions are defined for inventory locations within all shelf units within the shopping store and stored in map database 140 as a 3D map of the real space (shopping store). As shown in FIG. 3, coordinate locations along three axes can be used to calculate the length, depth, and height of an inventory location.

In one embodiment, the map identifies configurations of units of volume that correlate with portions of inventory locations on an inventory display structure within an area of real space. Each portion is defined by a start and end position along three axes of real space. A similar configuration of the inventory location portion can also be generated using a 2D map inventory location that divides the front view of the display structure.

[Joint data structure]

Image recognition engines 112a-112n receive the image sequence from camera 114 and process the images to generate corresponding arrays of joint data structures. The system includes processing logic that tracks the location of multiple objects (or customers in a shopping store) within an area of real space using image sequences generated by multiple cameras. In one embodiment, the image recognition engine 112a-112n identifies 19 possible objects in each element of the image that can be used to identify objects within areas that may be taking or placing inventory. identify one of the joints Possible joints can be classified into two categories: ankle joints and non-ankle joints. A nineteenth type of joint classification is for all non-joint features of the subject (ie, image elements that are not classified as joints). In other embodiments, the image recognition engine may be configured to specifically identify hand positions. We also use user check-in procedures or other techniques, such as biometric identification processes, for the purposes of identifying subjects and linking subjects with the detected positions of their hands as they move through the store. can be expanded.
Ankle:
Ankle joint (left and right)
Non-ankle:
neck
nose
eye (left and right)
ear (left and right)
shoulder (left and right)
Elbow (left and right)
wrist (left and right)
buttocks (left and right)
knee (left and right)
Non-joint

An array of joint data structures for a particular image groups elements of a particular image by joint type, time of the particular image, and coordinates of the element within the particular image. In one embodiment, the image recognition engines 112a-112n are convolutional neural networks (CNNs), the joint type is one of 19 types of joints of the subject, and the time of a particular image is determined by the source camera 114 for a particular image. The timestamp of the generated image, where the coordinates (x,y) specify the position of the element on the 2D image plane.

The output of the CNN is a matrix of confidence arrays for each image per camera. The matrix of confidence arrays is converted to an array of joint data structures. A joint data structure 400 as shown in FIG. 4 is used to store information for each joint. The joint data structure 600 identifies the x- and y-positions of elements within a particular image within the 2D image space of the camera from which the image is received. The joint number identifies the type of joint identified. For example, in one embodiment, the values range from 1-19. A value of 1 indicates that the joint is the left ankle, a value of 2 indicates that the joint is the right ankle, and so on. The joint type is selected using the confidence array for that element in the CNN's output matrix. For example, in one embodiment, the value of the joint number is "1" if the value corresponding to the left ankle joint is the highest in the confidence array for that image element.

The confidence number indicates how confident the CNN is in predicting that joint. A high confidence number value indicates that the CNN is confident in its predictions. To uniquely identify a joint data structure, an integer ID is assigned to the joint data structure. Following the above mapping, the output matrix 540 of per-image confidence arrays is transformed into an array of per-image joint data structures. In one embodiment, joint analysis includes performing a combination of k-nearest neighbors, Gaussian mixtures, and various image morphology transformations on each input image. The result contains an array of joint data structures that can be stored in the form of bitmasks in a ring buffer that maps the number of images at each time point to a bitmask.

[Subject Tracking Engine]

［被写体データ構造］ Tracking engine 110 is configured to receive an array of joint data structures generated by image recognition engines 112a-112n corresponding to images in an image sequence from cameras having overlapping fields of view. An array of joint data structures per image is sent to tracking engine 110 over network 181 by image recognition engines 112a-112n. The tracking engine 110 transforms the coordinates of the elements in the array of joint data structures corresponding to the various image sequences into candidate joints having coordinates in real space. A position in real space is covered by the fields of view of two or more cameras. The tracking engine 110 comprises logic for identifying a set of candidate joints having coordinates in real space (a constellation of joints) as an object in real space. In one embodiment, the tracking engine 110 accumulates an array of joint data structures from the image recognition engine for all cameras at a given time, so that it can be used to identify a constellation of candidate joints. This information is stored in the object database 140 as a dictionary. The dictionary can be organized in the form of key-value pairs, where the key is the camera ID and the value is an array of joint data structures from the camera. In such embodiments, this dictionary is used in a heuristics-based analysis to determine candidate joints and assign joints to subjects. In such an embodiment, the high-level inputs, processing, and outputs of tracking engine 110 are shown in Table 1. Details of the logic applied by the object tracking engine 110 that combines candidate joints to generate an object and track the movement of the object within an area of real space are described in US Pat. 853, "Object Recognition and Tracking Using Image Recognition Engines", which is incorporated herein by reference.

Table 1: Inputs, Processing, and Outputs from Object Tracking Engine 110 in an Exemplary Embodiment

[Subject data structure]

The object tracking engine 110 uses heuristics to connect the joints of the objects identified by the image recognition engines 112a-112. In doing so, the object tracking engine 110 creates new objects and updates the positions of existing objects by updating their joint positions. The object tracking engine 110 uses triangulation techniques to project joint positions from 2D space coordinates (x,y) to 3D real space coordinates (x,y,z). FIG. 5 shows an object data structure 500 for storing objects. The data structure 500 stores subject-related data as a key-value dictionary. The key is the frame ID and the value is another key-value dictionary, where the key is the camera ID and the value is a list of the 18 joints (of the subject) and their positions in real space. Subject data is stored in a subject database. Each new subject is also assigned a unique identifier that is used to access the subject's data in the subject database.

In one embodiment, the system identifies the joints of the subject and creates the skeleton of the subject. The skeleton is projected onto the real space and indicates the position and orientation of the subject in the real space. This is also called "pose estimation" in the field of machine vision. In one embodiment, the system displays the orientation and position of the object in real space on a graphical user interface (GUI). In one embodiment, the subject identification and image analysis is anonymous, ie, the unique identifier assigned to the subject produced by the joint analysis does not identify the subject's personal identifying information, as described above.

In this embodiment, a constellation of the identified subject's joints generated by time-series analysis of the joint data structure can be used to locate the subject's hand position. For example, the position of the hand of the identified subject can be identified using the position of the wrist joint alone, or the position based on the projection of the combination of the wrist and elbow joints.

[Inventory event]

FIG. 6 shows subject 240 removing inventory from a shelf in shelving unit B 204 in top view 610 of aisle 116a. The disclosed technique uses image sequences generated by at least two cameras in a plurality of cameras to locate inventory events. A single subject's joints may appear in the image frames of multiple cameras in each image channel. In the shopping store example, the subject moves through an area in real space, picks up items from inventory locations, and places items back into inventory locations. In one embodiment, the system predicts inventory events (also called put or take, plus or minus events) using a pipeline of convolutional neural networks called WhatCNN and WhenCNN.

A data set containing the objects identified by the joints in the object data structure and the corresponding image frames from the sequence of image frames per camera is provided as input to the bounding box generator. A bounding box generator implements logic that processes the dataset to specify a bounding box containing an image of an identified subject's hand in the images in the image sequence. The bounding box generator generates, for each camera, in each source image frame using, for example, the positions of the wrist joints (relative to each hand) and elbow joints in the articulated data structure 500 corresponding to each source image frame. identify the hand position of the In one embodiment, where the coordinates of the joints in the object data structure indicate the positions of the joints in 3D real space coordinates, the bounding box generator converts the joint positions from 3D real space coordinates to 2D Map to coordinates.

A bounding box generator creates bounding boxes for the hands in the image frames in the circular buffer for each camera 114 . In one embodiment, the bounding box is a 128 pixel (width) by 128 pixel (height) portion of the image frame, and the hand is centered in the bounding box. In other embodiments, the size of the bounding box is 64 pixels by 64 pixels or 32 pixels by 32 pixels. For m objects in an image frame from a camera, there can be up to 2m hands and thus 2m bounding boxes. However, in practice, less than 2m hands are visible in the image frame due to occlusion by other subjects or other objects. In one exemplary embodiment, the subject's hand position is inferred from the elbow and wrist joint positions. For example, the position of the subject's right hand is extrapolated as the extrapolation amount×(p2−p1)+p2 using the position of the right elbow (identified as p1) and the position of the right wrist (identified as p2). . Here the extrapolation amount is 0.4. In another embodiment, joint CNNs 112a-112n are trained using left and right hand images. Thus, in such embodiments, the joint CNNs 112a-112n directly identify the hand position within the image frame per camera. The hand positions per image frame are used by the bounding box generator to generate a bounding box for the identified hand.

WhatCNN is a convolutional neural network trained to process a specified bounding box in an image to generate a class of hands for identified subjects. One trained WhatCNN processes image frames from one camera. In the shopping store example embodiment, for each hand in each image frame, WhatCNN identifies whether the hand is empty. WhatCNN also determines the SKU (Stock Keeping Unit) number of the item in hand, the confidence value indicating the item in hand is a non-SKU item (i.e., does not belong to shopping store inventory), and the hand in the image frame. to identify the situation of the location of

The outputs of the WhatCNN models for all cameras 114 are processed by a single WhenCNN model for a given period of time. In the shopping store example, WhenCNN performs a time series analysis on the subject's hands to identify whether the subject is taking the store inventory item from the shelf or placing the store inventory item on the shelf. The disclosed technique uses image sequences generated by at least two of the plurality of cameras to locate inventory events. WhenCNN performs analysis of datasets from image sequences from at least two cameras to determine the location of inventory events in three dimensions and to identify items associated with inventory events. A time series analysis of WhenCNN's output per subject over time is performed to identify inventory events and their time of occurrence. A non-maximum suppression (NMS) algorithm is used for this purpose. When one inventory event (i.e., placing or picking up an item by a subject) is detected multiple times by WhenCNN (both from the same camera and multiple cameras), the NMS eliminates redundant events for the subject. NMS rescores include two main tasks: 'matching loss', which penalizes redundant detections, and neighborhood 'joint processing' to know if there are better detections at hand. ring technology.

The true take and put events for each subject are further processed by calculating the average of the SKU logits for the 30 image frames preceding the image frame with the true event. Finally, the maximum value argument (abbreviated as arg max or argmax) is used to determine the maximum value. Inventory items sorted by argmax values are used to identify inventory items that have been shelved or taken off the shelf. The disclosed technology attributes inventory events to subjects by assigning inventory-related inventory events to the subject's log data structure (or shopping cart data structure). Inventory items are added to a log of SKUs (also called shopping carts or baskets) for each subject. The image frame identifier "frame ID" of the image frame that led to detection of the inventory event is also stored with the identified SKU. The inventory event attributed logic matches the location of the inventory event with the location of one of the customers. For example, an image frame can be used to identify the 3D location of an inventory event represented by the subject's hand position at at least one point in the sequence that is classified as an inventory event using the subject data structure 500. , and can be used to determine the inventory location from where the item was picked or placed. The disclosed technique uses image sequences generated by at least two of the cameras to locate inventory events and create an inventory event data structure. In one embodiment, an inventory event data structure stores an item identifier, a put or take indicator, three-dimensional coordinates of an area in real space, and a timestamp. In one embodiment, inventory events are stored in inventory event database 150 .

The location of inventory events (placement and removal of inventory by a subject within an area of space) can be used to identify inventory locations, such as shelves, where the subject has picked or placed an item, in store planograms. Or you can compare it with other maps. Illustration 660 shows determining a shelf within a shelving unit by calculating the shortest distance from the hand position 640 associated with the inventory event. This shelf determination is then used to update the shelf inventory data structure. An exemplary inventory data structure 700 (also called a log data structure) is shown in FIG. This inventory data structure stores subject, shelf or store inventory as a key-value dictionary. The key is a unique identifier for an object, shelf or store and the value is another key-value dictionary where the key is a product identifier such as a stock keeping unit (SKU) and the value is an inventory event prediction. Along with the "frame ID" of the resulting image frame is a number that identifies the quantity of the item. A frame identifier (“frame ID”) can be used to identify the image frame that resulted in the identification of the inventory event that resulted in the association of the inventory item with the subject, shelf, or store. In other embodiments, a “camera ID” that identifies the source camera may be combined with the frame ID and stored within inventory data structure 700 . In one embodiment, the "frame ID" is the subject identifier, since the frame has the subject's hands within the bounding box. In other embodiments, other types of identifiers, such as "subject IDs" that explicitly identify the subject within an area of real space, can be used to identify the subject.

Once the shelf inventory data structure is integrated with the subject's log data structure, the shelf inventory is decremented to reflect the quantity of items the customer has picked up from the shelf. When a customer places an item on a shelf or an employee stocks an item on a shelf, the item is added to the inventory data structure for each inventory location. Over time, this process results in updating the shelf inventory data structure for all inventory locations within the shopping store. Integrate the inventory data structures of the inventory locations in the real space area to update the real space area inventory data structure showing the total number of items for each SKU in the store at that point in time. In one embodiment, such updates are performed after each inventory event. In another embodiment, the store inventory data structure is updated periodically.

Details of implementations of WhatCNN and WhenCNN that detect inventory events can be found in US patent application Ser. , which is incorporated herein by reference as if fully set forth herein.

[Real-time shelf and store inventory update]

FIG. 8 is a flowchart showing the processing steps for updating the shelf inventory structure within an area in real space. Processing begins at step 802 . At step 804, the system detects a take or put event within an area of real space. Inventory events are stored in inventory event database 150 . An inventory event record includes an item identifier, such as an SKU, a timestamp, and the location of the event within a three-dimensional area of real space indicating the location along three dimensions x, y, and z. Inventory events also include a put or take indicator to identify whether the subject has put the item on the shelf (also called a plus inventory event) or removed the item from the shelf (also called a minus inventory event). The inventory event information is combined with the output from object tracking engine 110 to identify the object associated with this inventory event. The results of this analysis are then used to update the subject log data structure (also called the shopping cart data structure) in inventory database 160 . In one embodiment, a subject identifier (eg, "subject ID") is stored in the inventory event data structure.

Using the subject's hand position (step 806) associated with the inventory event, the system can locate the nearest shelf within the inventory display structure (also referred to as the shelving unit above) in step 808. Store inventory engine 180 calculates hand distances to two-dimensional (2D) regions or areas on the xz plane (perpendicular to floor 220) of inventory locations within the shopping store. The 2D regions of inventory locations are stored in the shopping store's map database 140 . Let the hand be represented by a point E(x _event , y _event , z _event ) in real space. The shortest distance D from a point E in real space to any point P on the plane can be determined by projecting the vector PE onto the normal vector n to the plane. Existing mathematical techniques can be used to calculate the distance of the hand to all planes representing the 2D area of inventory locations.

In one embodiment, the disclosed technique includes calculating the distance from the location of the inventory event to the inventory location on the inventory display structure and matching the inventory event to the inventory location based on the calculated distance. A procedure is performed to match inventory event locations with inventory locations. For example, the closest inventory location (eg, shelf) from the location of the inventory event is selected and the inventory data structure for that shelf is updated in step 810 . In one embodiment, the position of the inventory event is determined by the position of the subject's hand along three coordinates in real space. If the inventory event is a take event (or a negative event) indicating that a bottle of ketchup has been taken by the subject, the shelf inventory is updated by decreasing the number of ketchup bottles by one. Similarly, if the inventory event is a put event indicating that the subject places a bottle of ketchup on the shelf, the number of ketchup bottles is incremented by one and the inventory on the shelf is updated. Similarly, the store's inventory data structure is updated accordingly. The quantity of the item placed in the inventory location is incremented by the same number in the store inventory data structure. Similarly, the quantity of items taken from the inventory location is subtracted from the store's inventory data structure in inventory database 160 .

At step 812, it is checked if a planogram is available for the shopping store or if it is known that a planogram is available. A planogram is a data structure that maps inventory items to inventory locations within a shopping store, which can be based on a plan for placement of inventory items within the store. If a planogram is available for the shopping store, at step 814 the items placed on the shelves by the subject are compared to the items on the shelves in the planogram. In one embodiment, the disclosed technology includes logic to determine misplaced items when an inventory event matches an inventory location that does not match the planogram. For example, if the SKU of the item associated with the inventory event matches the placement of the inventory item in the inventory location, then the item's location is correct (step 816), otherwise the item is misplaced. In one embodiment, at step 818, a notification is sent to the employee to remove the misplaced item from its current inventory location (eg, shelf) and move it to its correct inventory location according to the planogram. The system checks at step 820 whether the subject is leaving the shopping store using the speed, orientation and proximity to the store exit. If the subject is not leaving the store (step 820 ), processing resumes at step 804 . Otherwise, if the subject is determined to be leaving the store, then at step 822 the subject's log data structure (or shopping cart data structure) and the store's inventory data structure are merged.

In one embodiment, at step 810, if the items in the subject's shopping cart data structure are not deducted from the store inventory, the consolidation includes deducting those items from the store inventory data structure. In this step, the system may also identify items in the subject's shopping cart data structure that have low identification confidence scores and send notifications to store employees located near the store exit. The employee can then review the low identity confidence score items in the customer's shopping cart. This process does not require the store employee to compare all items in the customer's shopping cart to the customer's shopping cart data structure, only items with low confidence scores are sent to the store employee by the system. Identified and verified by store personnel. Processing ends at step 824 .

[Architecture for real-time shelf and store inventory updates]

An example architecture of a system in which customer inventory, inventory location (e.g., shelf) inventory, and store inventory (e.g., store-wide) data structures are updated using placing and picking items by customers within a shopping store: Shown in FIG. 9A. Since FIG. 9A is an architectural diagram, certain details have been omitted to improve the clarity of the description. The system shown in FIG. 9A receives image frames from multiple cameras 114 . As described above, in one embodiment, the cameras 114 can be temporally synchronized with each other such that images are acquired at the same time, or close in time and at the same image capture rate. Images acquired at all cameras covering an area of real space at the same time or close in time are assumed to represent different views at a point in time of an object having a fixed position in real space. Synchronized in the sense that they can be identified in the processing engine. Images are stored in a circular buffer 902 of image frames for each camera.

An “object identification” subsystem 904 (also referred to as a first image processor) processes image frames received from camera 114 to identify and track objects in real space. The first image processor includes a subject image recognition engine that detects joints of the subject in real space. The joints are connected to form an object and tracked as it moves in real space. Subjects are anonymous and tracked using an internal identifier "Subject ID".

A “Region Proposal” subsystem 908 (also called a third image processor), which includes a foreground image recognition engine, receives corresponding image sequences from multiple cameras 114 and identifies semantically significant objects in the foreground (i.e., The customer, the customer's hand, and the inventory) recognize the object as it relates to placing and taking inventory over time in the images from each camera. Region suggestion subsystem 908 also receives the output of object identification subsystem 904 . A third image processor processes the image sequences from camera 114 to identify and classify foreground changes represented in images in the corresponding image sequences. A third image processor processes the identified foreground changes to detect removal of inventory by the identified subject and placement of inventory on the inventory display structure by the identified subject. Create a detection set for . In one embodiment, the third image processor comprises a convolutional neural network (CNN) model, such as the WhatCNN mentioned above. The first set of detections is also called foreground detection of inventory put and take. In this embodiment, the output of WhatCNN is processed in a second convolutional neural network (WhenCNN) to determine the events of placing inventory items in inventory locations and the events of inventory placement within the inventory display structure by customers and store employees. create a first set of detections that identify take events for inventory items in . Details of the Region Proposal Subsystem are set forth in U.S. Patent Application Serial No. 15/907,112, entitled "Item Placement and Pickup Detection Using Image Recognition," filed February 27, 2018, which describes: is incorporated herein by reference as if fully set forth herein.

In another embodiment, the architecture for use in parallel with a third image processor to detect the placing and picking of inventory items and associate these placing and pickings with objects in the shopping store. a "semantic difference extraction" subsystem (also called a second image processor) that can The semantic difference extraction subsystem includes a background image recognition engine that receives corresponding image sequences from multiple cameras and extracts, for example, images in a background (i.e., an inventory display structure such as a shelf). Semantically significant differences are recognized when they relate to placing and taking inventory items over time in the images from each camera. A second image processor receives as inputs the output of object identification subsystem 904 and the image frames from camera 114 . Details of the "Semantic Differencing" subsystem can be found in US Pat. No. 15/945,473, filed May 4, entitled "Predicting Inventory Events Using Foreground/Background Processing," both of which are fully set forth herein. , incorporated herein by reference. A second image processor processes the identified background change to detect removal of inventory by the identified subject and placement of inventory on the inventory display structure by the identified subject. Create a detection set for . The second set of detections is also referred to as background detection of inventory put and take. In the shopping store example, the second detection identifies inventory items taken from an inventory location or placed on an inventory location by a customer or store employee. The semantic difference extraction subsystem includes logic for associating the identified background change with the identified object.

The system described in FIG. 9A includes selection logic for processing the first and second detection sets to generate a log data structure containing a list of inventory items for the identified subject. For putting and taking in real space, the selection logic selects the output from either the semantic difference extraction subsystem or the region proposal subsystem 908 . In one embodiment, the selection logic uses a confidence score generated by the semantic difference extraction subsystem for the first detection set and a confidence score generated by the region proposal subsystem for the second detection set. to make a selection. The output of the subsystem with the higher confidence score for a particular detection is selected and a log data structure 700 (also called a shopping cart data structure) containing a list of inventory items and their quantities associated with the identified subject. used to generate The shelf and store inventory data structures are updated using the subject log data structure as described above.

Object exit detection engine 910 determines whether the customer is moving towards the exit door and sends a signal to store inventory engine 190 . The store inventory engine determines whether one or more items in the customer's log data structure 700 have low identification confidence scores determined by the second or third image processors. If so, the inventory consolidation engine notifies the store employee located near the exit to confirm the items purchased by the customer. Objects, inventory locations, and shopping store inventory data structures are stored in inventory database 160 .

FIG. 9B shows a system in which customer inventory, inventory location (e.g., shelf) inventory, and store inventory (e.g., store-wide) data structures are updated using placing and taking items by customers within a shopping store. shows another architecture of Since FIG. 9A is an architectural diagram, certain details have been omitted to improve the clarity of the description. As described above, the system receives image frames from multiple synchronized cameras 114 . WhatCNN 914 uses an image recognition engine to determine items in a customer's hand within an area of real space (such as a shopping store). In one embodiment, there is one WhatCNN per camera 114 that performs image processing on the sequence of image frames produced by the respective camera. WhenCNN 912 performs a time series analysis of WhatCNN's output to identify put or take events. Inventory events are stored in database 918 along with item and hand information. This information is then combined with the customer information generated by the customer tracking engine 110 (also referred to above as the object tracking engine 110) by the person and product attributes component 920. FIG. A log data structure 700 of customers in the shopping store is generated by linking customer information stored in database 918 .

The disclosed technology uses a sequence of images generated by multiple cameras to detect customer departure from an area of real space. In response to detecting customer churn, the disclosed technology updates store inventory in memory for items associated with the customer-attributed inventory event. When exit detection engine 910 detects the exit of customer "C" from the shopping store, the items purchased by customer "C" as shown in log data structure 922 are integrated with store inventory data structure 924. , generates an updated store inventory data structure 926 . For example, as shown in FIG. 9B, a customer purchased two items 1, four items 3, and one item 4. The quantity of each item purchased by customer "C" as shown in customer log data structure 922 is subtracted from store inventory 924, reducing the quantity of item 1 from 48 to 46; Generate an updated store inventory data structure 926 indicating that the quantity of 3 and 4 has been reduced by the respective quantity of item 3 and item 4 purchased by customer "C". Customer “C” did not purchase Item 2, so in the updated store inventory data structure 926, the quantity of Item 2 remains the same as before in the current store inventory data structure 924.

In one embodiment, customer abandonment detection also triggers an update of the inventory data structure of the inventory location (eg, shopping store shelf) from which the customer picked up the item. In such embodiments, the inventory data structure for the inventory location is not updated immediately after the take or put inventory event, as described above. In this embodiment, when the system detects customer abandonment, the inventory events associated with the customer are traversed and linked to their respective inventory locations within the shopping store. This process updates the inventory data structure for the determined inventory location. For example, if a customer takes two items 1 from inventory location 27, the inventory data structure for inventory location 27 is updated by decreasing the quantity of item 1 by two. Note that inventory items may be stocked in multiple inventory locations within the shopping store. The system identifies the inventory location corresponding to the inventory event, and the inventory location from which the item is picked is updated accordingly.

[Store rearogram]

The location of inventory items throughout the physical space in the store, including the inventory locations in the shopping store, where the customer picks up the items from the inventory location and returns the items they don't want to buy to the same location on the same shelf from which the item was picked up. or change over time when the item is returned to a different location on the same shelf from which it was taken, or placed on a different shelf. The disclosed technology uses image sequences generated by at least two cameras in a plurality of cameras to identify inventory events and, in response to inventory events, locate inventory items within areas of real space. Chase. In some embodiments, items in a shopping store are arranged according to a planogram that identifies inventory locations (eg, shelves) where particular items are planned to be placed. For example, as shown in illustration 910 of FIG. 10, the left halves of shelves 3 and 4 are designated for merchandise (stocked in the shape of cans). Consider that at the beginning of the day or other inventory tracking interval (identified by time t=0), inventory locations are stocked according to the planogram.

The disclosed technique can compute a shopping store's "rearogram" at any time "t", which is a real-time map of the location of inventory items within an area of real space, which can be used in several implementations. The form can also be correlated with inventory locations within the store. Realogograms can be used to create a planogram by identifying inventory items and locations within a store and mapping them to inventory locations. In one embodiment, a system or method may create a data set defining a plurality of cells having coordinates within an area of real space. The system or method may use the length of the cell along the coordinates of the real space as an input parameter to divide the real space into data sets defining multiple cells. In one embodiment, cells are represented as a two-dimensional grid having coordinates within an area of real space. For example, the cells can be correlated with a 2D grid (eg, at 1-foot intervals) of front view inventory locations in a shelving unit (also called an inventory display structure), as shown in illustration 960 of FIG. As shown in FIG. 10, each grid is defined by its start position and end position on two-dimensional plane coordinates such as x-coordinate and z-coordinate. This information is stored in map database 140 .

In another embodiment, the cells are represented as a three-dimensional (3D) grid with coordinates within an area of real space. In one example, a cell can be correlated with a volume on an inventory location (or portion of an inventory location) of a shelf unit within a shopping store, as shown in FIG. 11A. In this embodiment, a map of real space identifies a configuration of units of capacity that can be correlated with portions of inventory locations on an inventory structure within an area of real space. This information is stored in map database 140 . Store realog engine 190 uses inventory event database 150 to calculate a realogogram of the shopping store at time “t” and stores it in realog database 170 . The shopping store re-alogram identifies the inventory items associated with the inventory events matched to the cell by their location at any time t by using the inventory event timestamps stored in the inventory event database 150. indicate. An inventory event includes an item identifier, a put or take indicator, the location of the inventory event represented by a position along three axes of an area in real space, and a timestamp.

The illustration in FIG. 11A shows that at the beginning of day 1 at t=0, the left portion of the inventory positions in the first shelf unit (forming the columnar array) contains "ketchup" bottles. Columns of cells (or grids) are shown in black in the schematic visualization, and cells can be rendered in other colors such as dark green. All other cells are left blank and not filled with a color indicating that they do not contain items. In one embodiment, a visualization of the items within the cell in the realogram is generated one item at a time showing its location (within the cell) within the store. In another embodiment, the realogram displays the location of a set of items on an inventory location using different colors to distinguish. In such embodiments, a cell may have multiple colors corresponding to several items associated with inventory events matching the cell. In another embodiment, other graphical or text-based visualizations are used to show the inventory items in the cell, such as by listing the SKUs or names in the cell.

The system uses each count of inventory events to calculate SKU scores (also called scores) during scoring for inventory items that have a matching position in a particular cell. The calculation of the score for a cell uses the total pick and put inventory weighted by the separation between the pick and put timestamp and the time of scoring. In one embodiment, the score is a weighted average of inventory events per SKU. In other embodiments, various scoring calculations may be used, such as counting inventory events per SKU. In one embodiment, the system displays the realogram as an image representing a cell within a plurality of cells and the score of the cell. For example, as an illustration in FIG. 11A, consider scoring time t=1 (eg, 1 day later). The re-alogram at time t=1 represents the scores for the "ketchup" product in different shades of black. In the store rearogram at time t=1, all four rows of the 1st shelf unit and the 2nd shelf unit (behind the 1st shelf unit) contain the "ketchup" product. Cells with higher SKU scores for "Ketchup" bottles are rendered darker gray compared to cells with lower scores for "Ketchup" bottles. Cells with zero score values for ketchup are not left blank and are not filled with color. Thus, the realogram presents real-time information about the location of ketchup bottles on inventory locations within the shopping store after time t=1 (eg, one day later). The generation frequency of the re-logogram can be set by the shopping store administration according to its requirements. Realog can also be generated on demand by store management. In one embodiment, the item location information generated by the rearogram is compared to the store planogram to identify misplaced items. Notifications can be sent to store personnel who can return misplaced inventory items to designated inventory locations as indicated in the store planogram.

In one embodiment, the system renders a display image representing a cell and the score of the cell within a plurality of cells. FIG. 11B shows a computing device with the realogram of FIG. 11A rendered on user interface display 1102 . Realog can be displayed on other types of computing devices such as tablets, mobile computing devices, and the like. The system can indicate the cell's score using color changes in the displayed image representing the cell. For example, in FIG. 11A, the column of cells containing "Ketchup" at t=0 can be represented by the dark green cells in that column. At t=1, the "ketchup" bottles are distributed over multiple cells beyond the first row of cells. The system can represent these cells by using different shades of green to indicate the cell's score. A darker shade of green indicates a higher score and a lighter green cell indicates a lower score. A user interface displays other generated information and provides tools for invoking and viewing functions.

[Calculation of store rearogram]

FIG. 12 is a flow chart presenting the processing steps for computing a re-alogram of shelves within an area of real space at time t, which may be adapted to other types of inventory display structures. Processing begins at step 1202 . At step 1204 , the system retrieves inventory events within the area of real space from the inventory event database 150 . An inventory event record includes an item identifier, a put or take indicator, the location of the inventory event represented by its position in three dimensions (such as x, y, and z) of an area in real space, and a timestamp. A put or pick indicator identifies whether a customer (also called a subject) has placed an item on the shelf or taken the item from the shelf. A put event is also called a positive inventory event and a take event is also called a negative inventory event. At step 1206, the inventory event is combined with the output from the subject tracking engine 110 to identify the subject's hand associated with this inventory event.

The system uses the subject's hand position (step 1206) associated with the inventory event to determine the position. In some embodiments, inventory events may be matched to the nearest shelf or other potential inventory location in a shelving unit or inventory display structure at step 1208 . Process step 808 in the flow chart of FIG. 8 details a technique that may be used to determine the position on the shelf that is closest to the hand position. As described in the method in step 808, the shortest distance D from a point E in real space to any point P on the plane (representing the front area of the shelf on the xz plane) is the vector PE. It can be determined by projecting onto the normal vector n to the plane. The intersection of vector PE with the plane gives the closest point on the shelf to the hand. The location of this point is stored in a "point cloud" data structure (step 1210) as a tuple containing the 3D location of the point in the area of real space, the SKU of the item, and the timestamp, the latter two being the inventory event • Obtained from a record. If there are more inventory event records (step 1211) in inventory event database 150, process steps 1204-1210 are repeated. If there are no more, processing proceeds to step 1214 .

The disclosed technique includes a data set stored in memory defining a plurality of cells having coordinates within an area of real space. A cell defines an area of real space bounded by a start position and an end position along a coordinate axis. An area of real space includes a plurality of inventory locations, and coordinates of cells within the plurality of cells can be correlated with inventory locations within the plurality of inventory locations. The disclosed technique matches the locations of inventory items associated with an inventory event with the coordinates of the cells and maintains data representing the cell-matched inventory items in a plurality of cells. In one embodiment, the system calculates the distance from the location of the inventory event to the cell in the dataset, and performs a procedure (step 808 in the flowchart of FIG. 8) for matching the inventory event to the cell based on the calculated distance. ) to determine the closest cell in the dataset based on the location of the inventory event. This matching of the event location with the closest cell provides the location of the point cloud data identifying the cell in which the point cloud data resides (step 1212). In one embodiment, a cell may map to a portion of an inventory location (such as a shelf) within an inventory display structure. Therefore, by using this mapping, the portion of the shelf is also identified. As mentioned above, a cell can be represented as a 2D grid or a 3D grid of areas in real space. The system includes logic that calculates scores during scoring for inventory items that have a matching position in a particular cell. In one embodiment, the score is based on counting inventory events. In this embodiment, the cell's score uses the sum of pick and put inventory weighted by the separation between the pick and put timestamp and the time of scoring. For example, the score can be a weighted moving average per SKU (also called SKU score), calculated for each cell using a "point cloud" of data points mapped to the cell:

The SKU score calculated by formula (1) is the sum of the scores of the data points of all point clouds of the SKU in the cell, where each data point is the number of days since the timestamp of the put and take events. Weighted by time point_t. Suppose there are two point cloud data points for the "ketchup" product in the grid. The first data point has a timestamp indicating that this inventory event occurred two days before the time "t" at which the realogram is calculated. Therefore, the value of point_t is "2". Point_t becomes "1" because the second data point corresponds to an inventory event that occurred one day before time "t". The ketchup score for a cell (identified by the cell ID that maps to the shelf identified by the shelf ID) is calculated as follows:

As the point cloud data points corresponding to inventory events age (ie, more days have passed since the event), their contribution to the SKU score decreases. At step 1216, the top 'N' SKUs are selected for the cell with the highest SKU score. In one embodiment, the system includes logic to select a set of inventory items for each cell based on the score. For example, a value of 'N' may be chosen as 10 to select the top 10 products per cell based on SKU score. In this embodiment, the realogram stores the top 10 products per cell. The updated realogram at time t is stored in step 1218 in realogram database 170 indicating the top "N" SKUs per cell in the shelf at time t. Processing ends at step 1220 .

In another embodiment, the disclosed technology does not use 2D or 3D maps of shelf portions stored in map database 140 to compute point cloud data on shelf portions corresponding to inventory events. . In this embodiment, a 3D physical space representing a shopping store is partitioned into cells represented as 3D cubes (eg, one-foot cubes). 3D hand positions are mapped to cells (using respective positions along three axes). The SKU score for all products is calculated cell by cell using equation (1) above. The resulting re-alogram shows the products in the cells in the real space representing the store, without requiring the locations of the shelves in the store. In this embodiment, the data points of the point cloud may be at the same position on coordinates in real space as the hand position corresponding to the inventory event, or may be closer to the hand position or closer to the hand position. It may be at a cell location within the area containing the location. This is because there may not be a shelf map and therefore the hand position is not mapped to the nearest shelf. Thus, the data points of the point cloud in this embodiment are not necessarily coplanar. All point cloud data points within a unit of volume in real space (eg, 1 cubic foot) are included in the calculation of the SKU score.

In some embodiments, a re-logogram is computed iteratively and used for temporal analysis of activity in the store, or an animation ( (such as stop motion animation).

[Example of application of store rearogram]

A store realogram can be used in many operations of a shopping store. Some applications of realograms are given in the following paragraphs.

[Restocking stock items]

FIG. 13A presents one such application of a store re-log for determining whether an inventory item needs to be restocked in an inventory location (such as a shelf). Processing begins at step 1302 . At step 1304, the system retrieves the realogram from the realogram database 170 at the scoring time "t". In one example, this is the most recently generated realogram. The SKU scores of all cells in the realogram are compared to a threshold score at step 1306 . If the SKU score is above the threshold (step 1308), the process repeats steps 1304 and 1306 for the next inventory item "i". In embodiments that include a planogram, or if a planogram is available, the SKU score of item "i" is compared to a threshold for cells that match the placement of inventory item "i" in the planogram. In another embodiment, an inventory item's SKU score is calculated by filtering "put on" inventory events. In this embodiment, the SKU score reflects the "take" event of inventory item "i" per cell in the realogram that can be compared to a threshold. In another embodiment, the count of "take" inventory events per cell can be used as a score for comparison with a threshold for determining the restocking of inventory item "i". In this embodiment, the threshold is the minimum count of inventory items that must be stocked in the inventory location.

If the SKU score for inventory item "i" is below the threshold, a warning notification is sent to the store manager or other designated employee indicating that inventory item "i" should be restocked (step 1310). ). The system can also identify inventory locations where inventory items need to be restocked by matching cells with SKU scores below a threshold to inventory locations. In another embodiment, the system checks the inventory level of inventory item "i" in the stock room of the shopping store to determine if inventory item "i" needs to be ordered from the wholesaler. can be done. Processing ends at step 1312 . FIG. 13B presents an exemplary user interface displaying restock warning notifications for inventory items. Alert notifications can be displayed on the user interface of other types of devices such as tablets and mobile computing devices. Alerts are sent to designated recipients via email, SMS (Short Message Service) on mobile phones, or notifications to save applications installed on mobile computing devices can also

[Misplaced Inventory Items]

In embodiments that include a planogram, or if a store planogram is otherwise available, for planogram compliance, the realog may identify misplaced items and thereby allow the planogram to be processed. is compared with In such embodiments, the system includes a planogram that specifies the placement of inventory items at inventory locations within an area of real space. The system includes logic to maintain data representing inventory items matching cells in a plurality of cells. The system determines misplaced items by comparing the data representing the inventory that matched the cell to the placement of the inventory items in the inventory locations specified in the planogram. FIG. 14 shows a flow chart for determining planogram compliance using a realogram. Processing begins at step 1402 . At step 1404, the system retrieves the realogram for inventory item "i" at scoring time "t". The scores of inventory item "i" in all cells in the realogogram are compared to the placement of inventory item "i" in the planogram (step 1406). If the rearogram shows SKU scores for inventory item "i" that exceed the threshold in cells that do not match the placement of inventory item "i" in the planogram (step 1408), the system identifies these items as misplacements. do. Warnings or notifications about items that do not match the placement of inventory items in the planogram are sent to store personnel who can take the misplaced items from their current locations and place them back in designated inventory locations. (Step 1410). If at step 1408 no misplaced items are identified, process steps 1404 and 1406 are repeated for the next inventory item "i".

In one embodiment, the store app displays the location of items on a store map and guides store employees to misplaced items. Following this, the store app can display the correct location of the item on the store map and guide the employee to the correct shelf section to place the item in the specified location. In another embodiment, the store app may direct customers to inventory based on shopping lists entered into the store app. Store apps can use the real-time location of inventory items using re-logograms to guide customers to the closest inventory location to an inventory item on a map. In this example, the closest location of an inventory item may be the location of a misplaced item that has not been placed in the inventory location according to the store's planogram. FIG. 14B shows an exemplary user interface displaying a misplaced inventory item “i” warning notification on user interface display 1102 . Various types of computing devices and alert notification mechanisms can be used to transmit this information to store personnel, as described above in FIG. 13B.

[Improved inventory product prediction accuracy]

Another application of Realog is to improve inventory prediction by image recognition engines. The flowchart of FIG. 15 illustrates exemplary processing steps for adjusting inventory forecasts using a realogram. Processing begins at step 1502 . At step 1504, the system receives the predicted confidence score probability for item "i" from the image recognition engine. As noted above, WhatCNN is an exemplary image recognition engine that identifies inventory items in the hands of a subject (or customer). WhatCNN outputs the confidence score (or confidence value) probability of the predicted inventory item. At step 1506, the confidence score probability is compared to a threshold. If the probability value exceeds the threshold, indicating greater confidence in the prediction (step 1508), the process is repeated for the next inventory item "i". Otherwise, if the confidence score probability is less than the threshold, processing continues at step 1510 .

The realogram for inventory item “i” at scoring time “t” is retrieved in step 1510 . In one example, this may be the most recent realoggram, in another example, a realoggram at a scoring time "t" that matches or is closer in time to the inventory event time is retrieved from the realoggram database 170. can be At step 1512, the SKU score of inventory item "i" at the location of the inventory event is compared to a threshold. If the SKU score is above the threshold (step 1514), accept the image recognition prediction for inventory item "i" (step 1516). Update the customer's log data structure associated with the inventory event accordingly. If the inventory event is a "take" event, add inventory item "i" to the customer's log data structure. If the inventory event is a "put" event, then remove inventory item "i" from the customer's log data structure. If the SKU score is below the threshold (step 1514), the image recognition engine prediction is rejected (step 1518). If the inventory event is a "take" event, then inventory item "i" is not added to the customer's log data structure as a result. Similarly, if the inventory event is "place", then inventory item "i" is not removed from the customer's log data structure as a result. Processing ends at step 1520 . In another embodiment, the SKU score of inventory item "i" can be used to adjust the input parameters to the image recognition engine for determining the item prediction confidence score. WhatCNN, a convolutional neural network (CNN), is an example of an image recognition engine that predicts inventory.

[Network configuration]

FIG. 16 shows the architecture of the network hosting the store realog engine 190 hosted on the network node 106 . The system includes a plurality of network nodes 101a, 101b, 101n, and 102 in the illustrated embodiment. In such embodiments, network nodes are also referred to as processing platforms. Processing platforms (network nodes) 103, 101a-101n, and 102, and cameras 1612, 1614, 1616, . A similar network hosts a store inventory engine 180 hosted on network node 104 .

FIG. 13 shows a plurality of cameras 1612, 1614, 1616, . . . 1618 connected to a network. Multiple cameras can be deployed in a particular system. In one embodiment, cameras 1612-1618 are connected to network 1681 using Ethernet-based connectors 1622, 1624, 1626, and 1628, respectively. In such embodiments, the Ethernet-based connector has a data transfer rate of 1 Gigabit/second, also called Gigabit Ethernet. It should be appreciated that in other embodiments, camera 114 is connected to the network using other types of network connections that may have faster or slower data transfer rates than Gigabit Ethernet. . Also, in alternate embodiments, a set of cameras can be directly connected to each processing platform, and the processing platforms can be coupled to a network.

Storage subsystem 1630 stores the basic programming and data structures that provide the functionality of certain embodiments of the present invention. For example, various modules that implement the functionality of store realog engine 190 may be stored in storage subsystem 1630 . Storage subsystem 1630 is an example of a computer readable memory comprising a non-transitory data storage medium and includes logic for computing a realogram of an area of real space according to the processes described herein. computer instructions stored in a computer-executable memory for performing all or any combination of data processing functions and image processing functions performed. In other examples, the computer instructions may be stored in other types of memory, including portable memory, including non-transitory computer-readable data storage media or media.

These software modules are generally executed by processor subsystem 1650 . The host memory subsystem 1632 typically includes a main random access memory (RAM) 1634 for storage of instructions and data during program execution, and a read only memory (ROM) 1636 in which fixed instructions are stored. contains several memories including In one embodiment, RAM 1634 is used as a buffer to store the point cloud data structure tuples generated by store realog engine 190 .

File storage subsystem 1640 provides persistent storage for program and data files. In one exemplary embodiment, storage subsystem 1640 includes four 120 gigabyte (GB) solid state disks (SSDs) in a RAID 0 (Redundant Array of Independent Disks) configuration identified by number 1642 . In the exemplary embodiment, data in map database 140, inventory event data in inventory event database 150, inventory data in inventory database 160, and realogram data in realogram database 170 not in RAM. Data is stored in RAID0. In the exemplary embodiment, hard disk drives 1646 have slower access speeds than RAID 0 1642 storage. A Solid State Disk (SSD) 1644 contains the operating system and related files for the Store Realog Engine 190 .

In the exemplary configuration, four cameras 1612 , 1614 , 1616 , 1618 are connected to processing platform (network node) 103 . Each camera has a dedicated graphics processing unit GPU1 1662, GPU2 1664, GPU3 1666, and GPU4 1668 to process the images sent by the camera. It is understood that less or more than three cameras can be connected per processing platform. Fewer or more GPUs are therefore configured in the network node such that each camera has a dedicated GPU for processing image frames received from the camera. Processor subsystem 1650 , storage subsystem 1630 , and GPUs 1662 , 1664 and 1666 communicate using bus subsystem 1654 .

Network interface subsystem 1670 is connected to bus subsystem 1654 forming part of processing platform (network node) 104 . Network interface subsystem 1670 provides interfaces to external networks, including interfaces to corresponding interface devices in other computer systems. The network interface subsystem 1670 allows the processing platform to communicate over a network using cables (or wires) or wirelessly. A number of peripheral devices such as user interface output devices and user interface input devices are also connected to bus subsystem 1654 forming part of processing platform 104 . These subsystems and devices are intentionally not shown in FIG. 13 to improve clarity of illustration. Bus subsystem 1654 is shown schematically as a single bus, but alternate embodiments of the bus subsystem may use multiple buses.

In one embodiment, the camera 114 has a resolution of 1288×964, a frame rate of 30 FPS, and a varifocal lens with a working distance of 300 mm to infinity, 98.2° to 23.5°, at 1.3 megapixels/image. It can be implemented using a Chameleon3 1.3 MP Color USB3 Vision (Sony ICX445) with a 1/3 inch sensor field of view of 8°.

The technology described herein also includes a system for tracking inventory items within an area of real space that includes an inventory display structure, along with corresponding methods and computer program products, to track inventory items above the inventory display structure. A plurality of cameras or other sensors positioned and sensors within the plurality of sensors that generate respective image sequences of inventory display structures within corresponding fields of view in real space, where the field of view of each sensor comprises a plurality of sensors. a memory overlapping the field of view of at least one other sensor in the sensor and storing a data set, wherein the data set defines a plurality of cells having coordinates within an area of real space; a processing system coupled to process the image sequences generated by the plurality of sensors to locate the inventory event in three dimensions within an area of real space; including logic for determining the closest cell in the data set based on the location of the inventory event, the processing system using the count of each of the inventory events to associate inventory events with locations that match the particular cell. Contains the logic to calculate the score when scoring for an inventory item. The system can include logic to select a set of inventory items for each cell based on the score. An inventory event can include an item identifier, a put or take indicator, a position represented by a position along three axes of an area in real space, and a timestamp. The system can include a data set defining a plurality of cells represented as a two-dimensional grid having coordinates within an area of real space, the cells correlated with front view portions of inventory locations, the processing system comprising: Contains logic to determine the closest cell based on inventory event location. The system can include a data set defining a plurality of cells represented as a three-dimensional grid having coordinates within an area of real space, the cells correlated with portions of the volume over inventory locations, the processing system comprising: Contains logic to determine the closest cell based on inventory event location. A put indicator can identify when an item has been placed in an inventory location, and a put indicator identifies when an item has been removed from an inventory location. Logic for processing image sequences generated by multiple sensors includes an image recognition engine that generates datasets representing elements in images corresponding to the hand, and analyzing datasets from the image sequences from at least two sensors. can be performed to determine the location of the inventory event in three dimensions. Image recognition engines may include convolutional neural networks. The logic for calculating the cell score uses the total put and take inventory weighted by the separation between the put and take time stamps and the time of scoring and stores the score in memory. be able to. The logic for determining the closest cell in the dataset based on the location of the inventory event consists of calculating the distance from the location of the inventory event to the cell in the dataset and adding the inventory event to the cell based on the calculated distance. can perform a procedure including matching with

Any data structures and code referred to above or above may be any device or medium capable of storing code and/or data for use by a computer system, according to many embodiments. computer readable memory, including any computer readable storage medium. This includes volatile memory, non-volatile memory, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), disk drives, magnetic tapes, CDs (compact disks), DVDs (digital versatile (including, but not limited to, magnetic and optical storage devices such as discs or digital video discs) or other media capable of storing computer readable media now known or hereafter developed.

The preceding description is presented to enable the use and practice of the disclosed techniques. Various modifications to the disclosed embodiments will be apparent, and the principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of the disclosed technology. Accordingly, the disclosed technology is not intended to be limited to the embodiments shown, but is to be accorded the broadest scope consistent with the principles and features disclosed herein. The scope of the disclosed technology is defined by the appended claims.

Claims (24)

A system for tracking inventory items within an area of real space, comprising:
a plurality of sensors and a processing system coupled to the plurality of sensors;
A sensor of the plurality of sensors produces a respective image sequence of a corresponding field of view in the real space, wherein the field of view of each sensor is matched with the field of view of at least one other sensor in the plurality of sensors. overlapping,
The processing system uses the image sequences generated by at least two of the plurality of sensors to identify an inventory event and, in response to the inventory event, determines inventory within an area of the real space. Logic for tracking the location of an item and matching the location of an inventory item to the coordinates of a cell in a plurality of cells having coordinates within an area of said real space to represent an inventory item that matches a cell in said plurality of cells. A system that includes logic to maintain data and, when scoring, for an inventory item that has a matching position in a particular cell, calculate a score based on inventory event counts. 2. The system of claim 1, wherein the inventory event includes an item identifier, a put or take indicator, a position represented by a position in three dimensions of the real space area, and a timestamp. the area of real space includes a plurality of inventory locations;
2. The system of claim 1 , including logic for matching inventory locations with coordinates of cells within a plurality of cells. Said logic for calculating the score of a cell claims to use said put and take totals weighted by the separation between said put and take time stamps and said scoring time. Item 1. The system according to Item 1 . 2. The system of claim 1 , wherein the processing system includes logic for rendering display images representing cells in the plurality of cells and the scores of the cells. 2. The system of claim 1 , wherein the processing system includes logic for selecting a set of inventory items per cell based on the score. the area of real space includes a plurality of inventory locations, the coordinates of cells within the plurality of cells are correlated with inventory locations within the plurality of inventory locations;
including in memory a planogram specifying placement of inventory items at inventory locations within the area of the real space;
Logic maintaining the data representing inventory items that match cells in the plurality of cells aligns the data representing inventory items that match cells with placement of inventory items within specified inventory locations within the planogram. 2. The system of claim 1 , including logic for determining misplaced items by comparison. the area of real space includes a plurality of inventory locations, the coordinates of cells within the plurality of cells are correlated with inventory locations within the plurality of inventory locations;
logic maintaining said data representing inventory items matching cells within said plurality of cells, wherein for a particular inventory item matched with a particular cell, the count of said particular inventory item within that cell determines said 2. The system of claim 1 , including logic for determining whether the inventory on an inventory location correlated with a particular cell is below a threshold for restocking. A method , executed by a computer system, for tracking inventory within an area of real space, comprising:
using a plurality of sensors to generate an image sequence of each corresponding said field of view in said real space, wherein the field of view of each sensor overlaps said field of view of at least one other sensor;
identifying inventory events using the image sequences generated by at least two of the plurality of sensors;
tracking the location of inventory items within the real space area in response to the identification of the inventory event;
matching locations of inventory items to coordinates of cells in a plurality of cells having coordinates within an area of the real space; and maintaining data representing inventory items that match cells in the plurality of cells; as well as,
A method comprising calculating a score using a count of inventory events during scoring for inventory items that have a matching position in a particular cell. 10. The method of claim 9 , wherein the inventory event includes an item identifier, a put or take indicator, a position represented by a position along three axes of the real space area, and a timestamp. the area of real space includes a plurality of inventory locations;
10. The method of claim 9 , wherein coordinates of cells within said plurality of cells correlate to inventory locations within said plurality of inventory locations. calculating a cell's score using the sum of the put and take inventory weighted by the separation between the put and take take time stamps of the inventory item and the time of the scoring; and 10. The method of claim 9 , further comprising: storing the score in memory. 10. The method of claim 9 , further comprising rendering a display image representing a cell in the plurality of cells and the score of the cell. 10. The method of claim 9 , further comprising selecting a set of inventory items for each cell based on said score. the area of real space includes a plurality of inventory locations, the coordinates of cells within the plurality of cells are correlated with inventory locations within the plurality of inventory locations ;
designating as a planogram the arrangement of inventory items at inventory locations within the real space area, and storing the planogram in memory;
10. The method of claim 9 , further comprising determining misplaced items by comparing the data representing inventory items that match a cell to placement of inventory items within the inventory locations. the area of real space includes a plurality of inventory locations, the coordinates of cells within the plurality of cells are correlated with inventory locations within the plurality of inventory locations;
For a particular inventory item that matched a particular cell, the count of said particular inventory item in that cell is below a threshold for restocking said inventory item on an inventory location correlated with said particular cell. 10. The method of claim 9, further comprising determining whether. A non-transitory computer-readable storage medium storing computer program instructions for tracking inventory items within an area of real space, comprising:
A method, implemented when said instructions are executed on a processor, comprising:
using a plurality of sensors to generate an image sequence of each corresponding said field of view in said real space, wherein the field of view of each sensor overlaps said field of view of at least one other sensor;
identifying inventory events using the image sequences generated by at least two of the plurality of sensors ;
tracking the location of inventory items within the real space area in response to the identification of the inventory event ;
matching locations of inventory items to coordinates of cells in a plurality of cells having coordinates within an area of the real space; and maintaining data representing inventory items that match cells in the plurality of cells; as well as,
calculating a score using a count of inventory events during scoring for inventory items having a location matching a particular cell. 18. The non-transitory computer-readable storage medium of claim 17 , wherein the inventory event comprises an item identifier, a put or take indicator, a position represented by a position along three axes of the real space area, and a timestamp. . the area of real space includes a plurality of inventory locations;
18. The non-transitory computer-readable storage medium of claim 17 , wherein coordinates of cells within said plurality of cells correlate to inventory locations within said plurality of inventory locations. Carrying out the method comprises:
calculating a cell's score using the sum of the put and take inventory weighted by the separation between the put and take take time stamps of the inventory item and the time of the scoring; and 18. The non-transitory computer-readable storage medium of claim 17 , further comprising: storing the score in memory. Carrying out the method comprises:
18. The non-transitory computer-readable storage medium of claim 17 , further comprising rendering a display image representing a cell in the plurality of cells and the score of the cell. Carrying out the method comprises:
18. The non-transitory computer-readable storage medium of claim 17 , further comprising selecting a set of inventory items for each cell based on said score. the area of real space includes a plurality of inventory locations, the coordinates of cells within the plurality of cells are correlated with inventory locations within the plurality of inventory locations;
Carrying out the method comprises :
designating as a planogram the arrangement of inventory items at inventory locations within the real space area, and storing the planogram in memory;
18. The non-transitory computer readable of claim 17 , further comprising determining misplaced items by comparing the data representing inventory items that match a cell to placement of inventory items within the inventory location. storage medium. the area of real space includes a plurality of inventory locations, the coordinates of cells within the plurality of cells are correlated with inventory locations within the plurality of inventory locations;
Carrying out the method comprises:
For a particular inventory item that matched a particular cell, the count of said particular inventory item in that cell is below a threshold for restocking said inventory item on an inventory location correlated with said particular cell. 18. The non-transitory computer-readable storage medium of claim 17, further comprising determining whether.

Images