JP2021533449A - Store realogram based on deep learning - Google Patents

Abstract

Systems and methods for tracking inventories in real-space areas are provided. Multiple cameras or other sensors generate their respective image sequences within the corresponding fields of view in real space. The field of view of each camera overlaps the field of view of at least one other camera. The system is coupled to multiple cameras and uses an image sequence generated by at least two cameras in the multiple cameras to identify inventory events. Inventory events include product identifiers, locations, and time stamps. A plurality of cells having coordinates in an area of real space are stored in memory as a data set. The processing system uses each count of inventory events to calculate a score at the time of scoring for an inventory item that has a position that matches a particular cell. [Selection diagram] FIG. 11A

Description

Priority application

This application is incorporated herein by reference in US Provisional Patent Application No. 62 / 703,785 (agent reference number STCG 1006-1), filed July 26, 2018, and January 24, 2019. Claim the interests of US Patent Application No. 16 / 256,355 (agent reference number STCG 1007-1). The US Patent Application No. 16 / 256,355 claims the benefit of US Provisional Patent Application No. 62 / 542,077 (agent reference number STCG 1000-1) filed August 7, 2017, 2017. A partial continuation of US Patent Application No. 15 / 847,796 (agent reference number STCG 1001-1) (currently US Patent No. 10,055,853, issued on August 21) filed on December 19th. There is a US Patent Application No. 15 / 907,112 filed on February 27, 2018 (agent reference number STCG 1002-1) (currently US Patent No. 10,133,933, issued November 20, 2018). ), Which is a partial continuation application of US Patent Application No. 15 / 945,473 (agent reference number STCG 1005-1) filed on April 4, 2018. These US patent applications are incorporated herein by reference.

The present invention relates to a system for tracking inventory items in a real space area including an inventory display structure.

Determining the quantity and location of various inventories stocked in an inventory display structure within a real space area such as a shopping store is required for the efficient operation of the shopping store. A subject in a real-space area, such as a customer, takes a product from a shelf and places the product in their shopping cart or basket. Customers can also put the item back on the same shelf or on a different shelf if they do not want to purchase the item. Thus, over a period of time, inventories may be removed from a designated location on a shelf and distributed to other shelves in a shopping store. In some systems, the quantity of goods in stock is available after a significant delay because the receipt and stock inventory need to be linked. Delays in the availability of information about the quantity of merchandise stocked in a shopping store can affect customer purchase decisions as well as store manager behavior to order more high-demand inventory merchandise. be.

It is desirable to provide a system that can more effectively and automatically provide the amount of goods stocked on the shelves in real time and identify the position of the goods on the shelves.

A system for tracking inventory items in a real space area and a method of operating the system are provided. Multiple cameras or other sensors generate an image sequence for each of the corresponding fields of view in real space. The system includes processing logic that is coupled to multiple sensors and uses an image sequence generated by at least two sensors to identify inventory events. The system tracks inventory items in a real-space area in response to inventory events.

A system and method for tracking inventory items in a real space area are provided. Multiple cameras or other sensors generate an image sequence for each of the corresponding fields of view in real space. The field of view of each sensor overlaps with the field of view of at least one other sensor in the plurality of sensors. The system uses a sequence of images generated by at least two sensors in multiple sensors to identify inventory events. The system tracks the location of inventory items within a real-space area in response to inventory events.

In one embodiment, the inventory event comprises a goods identifier, an indicator to place or take, a position represented by a position along three axes of an area in real space, and a time stamp. The system may include or have access to a memory that stores a dataset that defines multiple cells with coordinates within an area of real space. The system includes logic that matches the position of the inventory item with the coordinates of the cell, and maintains data representing the inventory item that matches the cells in a plurality of cells. An area in real space can contain multiple inventory locations. The coordinates of a cell within a plurality of cells can correlate with a stock position or part of a stock position within a plurality of stock positions. The system includes logic to calculate the score at the time of scoring for inventory items that have positions that match a particular cell, using each count of inventory events. The logic for calculating the cell's score uses the sum of placing and taking inventory items weighted by the separation between the putting and taking timestamps and the scoring time. The system contains logic to store the score in memory.

In one embodiment, the system comprises a logic that renders a cell in a plurality of cells and a display image representing the score of the cell. In this embodiment, the score is expressed by the change in color in the display image representing the cell. The system includes logic to select a set of inventories per cell based on the score. In one embodiment, the real space area contains a plurality of inventory positions, and the coordinates of the cells in the plurality of cells correlate with the inventory positions in the plurality of inventory positions. In this embodiment, the dataset in memory defines a plurality of cells having coordinates within an area of real space.

The system may include or have access to a memory that stores a stock position within an area of real space and a planogram that identifies the stock item placed at the stock position. The planogram can also contain information about the portion of the stock position specified for a particular stock item. Planograms can be generated based on a plan for the placement of inventory goods on inventory locations within real-space areas.

The system includes logic that maintains data representing in-stock items that match cells within multiple cells. The system can also include logic to determine misplaced merchandise by comparing data representing in-stock merchandise that matches the cell with the planogram.

As discussed herein, the system identifies the location of an inventory item within a real-space area based on the accumulation of data about the item and its position in the detected inventory event, "rear" herein. A data structure called a "loggram" can be generated and stored in memory. The data in the realogram is compared to the data in the planogram to determine how the in-stock items are placed in the area compared to the plan, such as finding the location of misplaced items. be able to. Realograms are also used to locate inventory items in 3D cells and correlate those cells with inventory positions in stores, for example, as can be determined from inventory position planograms or other maps. Can be processed. You can also process the realogram to track activities related to specific inventory items at various locations within the area. Other uses of rearograms are also possible.

Systems and methods for tracking inventory items within a real-space area, including inventory display structures, are provided. The system includes multiple cameras located above the inventory display structure. The camera produces each image sequence of the inventory display structure in the corresponding field of view in real space. The field of view of each camera overlaps the field of view of at least one other camera in the plurality of cameras. A dataset defines multiple cells that have coordinates within an area of real space. The dataset is stored in memory. The system processes an image sequence generated by multiple cameras to locate inventory events in three dimensions within an area of real space. In response to an inventory event, the system contains logic to determine the closest cell in the dataset based on the location of the inventory event. The system includes logic that calculates the score at the time of scoring for inventory items related to an inventory event that have positions that match a particular cell using each count of the inventory event.

In one embodiment, the system includes logic to select a set of inventories per cell based on a score. In one embodiment, the inventory event comprises a goods identifier, an indicator to place or take, a position represented by a position along three axes of an area in real space, and a time stamp. In one embodiment, the system comprises a dataset that defines multiple cells represented as a two-dimensional grid with coordinates within an area of real space. The cell can correlate with a portion of the front view of the stock position. The processing system includes logic to determine the closest cell based on the position of the inventory event. In one embodiment, the system comprises a dataset that defines multiple cells represented as a three-dimensional grid with coordinates within an area of real space. The cell can correlate with a portion of the volume on the stock position. The processing system includes logic to determine the closest cell based on the position of the inventory event. The put indicator identifies that the item has been placed in the inventory position, and the take indicator identifies that the item has been removed from the inventory position.

In one embodiment, the logic for processing an image sequence generated by a plurality of cameras comprises an image recognition engine. The image recognition engine produces a dataset that represents the elements in the image that correspond to the hand. The system includes logic that performs analysis of the dataset from image sequences from at least two cameras to determine the location of inventory events in three dimensions. The image recognition engine comprises a convolutional neural network.

In one embodiment, the system uses the sum of taking and putting in stock weighted by the separation between the putting and taking time stamps and the scoring time to calculate the cell score. including. The score is stored in memory. In one embodiment, the logic that determines the closest cell in the dataset based on the location of the inventory event is to calculate the distance from the location of the inventory event to the cell in the dataset and based on the calculated distance. Includes matching inventory events with cells.

Methods and computer program products that can be performed by a computer system are also described herein.

The functions described herein identify inventory events, including goods related to inventory events, link to cells within multiple cells that have coordinates within an area of real space, and store realograms. With respect to, but not limited to, the type of image data to be processed, which processing of the image data to perform, and how to reliably determine the behavior from the image data, including, but not limited to, updating. Presents the complex problems of computer engineering.

Other aspects and advantages of the invention can be understood by examining the following drawings, detailed description, and claims.

Shown is an architectural level schematic of a system in which a store inventory engine and a store realogram engine track inventory items in a real space area including an inventory display structure.

It is a side view of the aisle in a shopping store which shows the subject in a shopping store, the inventory display structure, and the arrangement of a camera.

It is a perspective view of the inventory display structure in the aisle of FIG. 2A which shows the subject which takes out the goods from the shelf in the inventory display structure.

An example of a 2D and 3D map of a shelf in an inventory display structure is shown.

An exemplary data structure for storing joint information of a subject is shown.

An exemplary data structure for storing a subject, including information on related joints, is shown.

It is a top view of the inventory display structure of the shelf unit in the aisle of FIG. 2A in a shopping store showing the selection of shelves in the inventory display structure based on the position of the inventory event indicating the goods taken out of the shelves.

Shown is an example of a log data structure that can be used to store a subject's shopping cart or inventory items that are stocked on a shelf or in a shopping store.

It is a flowchart which shows the process step which determines the inventory product on a shelf and in a shopping store based on the position of placing and taking an inventory product.

It is an exemplary architecture in which the techniques shown in the flowchart of FIG. 8 can be used to determine inventory items on shelves within an area of real space.

It is an exemplary architecture in which the store inventory data structure can be updated using the techniques shown in the flowchart of FIG.

Shown is the discretization of shelves in a portion of an inventory display structure using a two-dimensional (2D) grid.

Inventory distributed from a specified location on a portion of a shelf in an inventory display structure to another location on the same shelf, and one day later to a position on a different shelf in another inventory display structure in the shopping store. It is an example of a realogram using a three-dimensional (3D) grid of shelves showing the location of goods.

It is an example showing the realogram of FIG. 11A displayed on the user interface of a computing device.

It is a flowchart which shows the process step for calculating the realogram of the inventory goods which are stocked in the shelf of the inventory display structure in a shopping store based on the position of putting and taking stock goods.

It is a flowchart which shows the processing step which determines the restocking of an inventory product using a realogram.

An exemplary user interface for displaying restock notifications for in-stock items.

It is a flowchart which shows the processing step which determines the compliance of a planogram using a realogram.

An exemplary user interface for displaying misplaced merchandise notifications for in-stock merchandise.

It is a flowchart which shows the processing step for adjusting the confidence score probability of an inventory product forecast using a realogram.

A camera and computer hardware configuration configured to host the inventory integration engine and store realogram engine of FIG.

The following description is presented to allow one of ordinary skill in the art to create and use the invention and is provided in line with the particular application and requirements thereof. Various modifications to the disclosed embodiments will be readily apparent to those of skill in the art, and the general principles defined herein do not deviate from the spirit and scope of the invention of other embodiments and uses. Can be applied to. Accordingly, the invention is not intended to be limited to the embodiments shown, and should be given the broadest scope consistent with the principles and features disclosed herein.
[System overview]

The system of the target technique and various embodiments will be described with reference to FIGS. 1 to 13. The system and processing will be described with reference to FIG. 1, which is a schematic of the architecture level of the system according to this embodiment. Since FIG. 1 is an architectural diagram, certain details are omitted to improve the clarity of the description.

The description of FIG. 1 is organized as follows. First, the elements of the system will be described, and then their interconnections will be described. Next, the use of elements in the system will be described in more detail.

FIG. 1 provides an explanatory diagram at the block diagram level of the system 100. The system 100 includes a camera 114, image recognition engines 112a, 112b and 112n hosted by a network node, a store inventory engine 180 located within a network node (or node) 104 on the network, and a network on the network. Store realogram engine 190 located within node (or node) 104, network node 102 hosting subject tracking engine, map database 140, inventory event database 150, planogram and inventory database 160, realogram -Includes a database 170 and one or more communication networks 181. The network node can host only one image recognition engine or multiple image recognition engines as described herein. The system can also include a subject database and other support data.

As used herein, a network node is addressable, connected to a network and capable of transmitting, receiving, or forwarding information to and from other network nodes over a communication channel. It is a hardware device or a virtual device. Examples of electronic devices that can be deployed as hardware network nodes include all types of computers, workstations, laptop computers, handheld computers, and smartphones. Network nodes can be implemented in cloud-based server systems. Multiple virtual devices configured as network nodes can be implemented using a single physical device.

For clarity, only three network nodes hosting the image recognition engine are shown in system 100. However, any number of network nodes hosting the image recognition engine can be connected to the subject tracking engine 110 via the network 181. Similarly, the image recognition engine, subject tracking engine, store inventory engine, store realogram engine, and other processing engines described herein run using multiple network nodes in a distributed architecture. be able to.

Next, the interconnection of the elements of the system 100 will be described. The network 181 is a network node 101a, 101b, and 101n that hosts the image recognition engines 112a, 112b, and 112n, respectively, a network node 104 that hosts the store inventory engine 180, and a network node that hosts the store realogram engine 190. The node 106, the network node 102 that hosts the tracking engine 110, the map database 140, the inventory event database 150, the inventory database 160, and the realogram database 170 are combined. The camera 114 is connected to the subject tracking engine 110 via a network node that hosts the image recognition engines 112a, 112b, and 112n. In one embodiment, the camera 114 is installed in a shopping store (supermarket, etc.), and a set (two or more) of cameras 114 having overlapping fields of view is arranged on each aisle to acquire an image of the real space in the store. do. In FIG. 1, two cameras are arranged on the passage 116a, two cameras are arranged on the passage 116b, and three cameras are arranged on the passage 116n. The camera 114 is installed on a passageway having overlapping fields of view. In such an embodiment, the camera is configured with the goal of having a customer moving in the aisle of a shopping store in the field of view of two or more cameras at any given time.

The cameras 114 can be temporally synchronized with each other so that the images are acquired simultaneously or close in time and at the same image capture rate. The camera 114 can send each continuous image stream at a predetermined rate to the network node hosting the image recognition engines 112a-112n. Images acquired by all cameras covering an area of real space at the same time or close in time are such that the synchronized image represents a different view of the subject with a fixed position in real space in the processing engine. Synchronized in the sense that they can be identified. For example, in one embodiment, the camera sends image frames at a rate of 30 frames per second (fps) to each network node hosting the image recognition engines 112a-112n. Each frame has a time stamp, camera identification information (abbreviated as "camera ID"), and frame identification information (abbreviated as "frame ID") together with image data. Other embodiments of the disclosed techniques use various types of sensors such as infrared image sensors, high frequency image sensors, ultrasonic sensors, thermal sensors, lidars, etc. to generate this data. be able to. In addition to the camera 114 that produces RGB color output, multiple types of sensors may be used, including, for example, infrared or high frequency image sensors. Multiple sensors are temporally synchronized with each other so that frames are captured simultaneously or temporally by the sensors at the same frame capture rate. In all embodiments disclosed herein, sensors other than cameras, or multiple types of sensors, may be used to generate the image sequence used.

Cameras installed on the aisle are connected to their respective image recognition engines. For example, in FIG. 1, the two cameras installed on the passage 116a are connected to the network node 101a hosting the image recognition engine 112a. Similarly, the two cameras installed on the passage 116b are connected to the network node 101b that hosts the image recognition engine 112b. Each image recognition engine 112a-112n hosted within the network nodes 101a-101n separately processes image frames received from one camera in the illustrated example.

In one embodiment, each image recognition engine 112a, 112b, and 112n is implemented as a deep learning algorithm such as a convolutional neural network (abbreviated as CNN). In such an embodiment, the CNN is trained using the training database 150. In the embodiments described herein, image recognition of a subject in real space is based on identifying and grouping recognizable joints in an image, where the group of joints belongs to an individual subject. be able to. For this joint-based analysis, the training database 150 collects enormous images for each of the different types of joints for the subject. In an exemplary embodiment of a shopping store, the subject is a customer moving through an aisle between shelves. In an exemplary embodiment, during CNN training, the system 100 is referred to as a "training system". After training the CNN using the training database 150, the CNN is switched to production mode to process the customer's image in the shopping store in real time.

In an exemplary embodiment, during production, the system 100 is referred to as a run-time system (also referred to as an inference system). The CNN of each image recognition device produces an array of joint data structures for the images in each image stream. In the embodiments described herein, an array of joint data structures is generated for each processed image so that each image recognition engine 112a-112n produces an output stream of the array of joint data structures. Generate. These arrays of joint data structures from cameras with overlapping fields of view are further processed to form groups of joints and identify such groups of joints as subjects. The system can identify and track a subject using the identifier "subject ID" while the subject is in an area of real space.

The subject tracking engine 110 is hosted on the network node 102 and, in this example, receives a continuous stream of an array of subject joint data structures from the image recognition engines 112a-112n. The subject tracking engine 110 processes an array of joint data structures and converts the coordinates of the elements in the array of joint data structures corresponding to the images of various sequences into candidate joints having coordinates in real space. For each set of synchronized images, the combination of candidate joints identified throughout real space can be considered to resemble a galaxy of candidate joints for analogical purposes. At each subsequent time point, the movement of the candidate joint is recorded as the galaxy changes over time. The subject tracking engine 110 identifies a subject in a real space area at a given point in time.

The tracking engine 110 uses logic that identifies a group or set of candidate joints with coordinates in real space as subjects in real space. For analogical purposes, each set of candidate points resembles the constellation of the candidate joint at each point in time. Candidate joint constellations can move over time. A time series analysis of the output of the subject tracking engine 110 over a period of time identifies the movement of the subject within an area of real space.

In an exemplary embodiment, the logic for identifying a set of candidate joints comprises a heuristic function based on the physical relationship between the joints of the subject in real space. These heuristic functions are used to identify a set of candidate joints as a subject. A set of candidate joints is within a given set of individual candidate joints that have heuristic parameter-based relationships with other individual candidate joints and that can be identified or can be identified as individual subjects. Includes a subset of candidate joints.

In the example of a shopping store, a customer (also referred to as the subject above) travels in an aisle and an open space. The customer retrieves the goods from the stock position on the shelf in the stock display structure. In one example of an inventory display structure, shelves are placed at various levels (or heights) from the floor, and inventory items are stocked on the shelves. The shelves may be fixed to the wall or placed as self-supporting shelves forming aisles within the shopping store. Other examples of inventory display structures include pegboard shelves, magazine shelves, rotary shelves, warehouse shelves, and refrigerated shelves units. Inventory items can also be stocked in other types of inventory display structures such as stacked wire baskets, dump bins and the like. Customers can also return merchandise from the shelves from which they were taken out to the same shelf or to another shelf.

The system includes a store inventory engine 180 (hosted on network node 104) to update inventory at inventory locations within a shopping store when a customer places an item on a shelf and removes the item from the shelf. .. The store inventory engine updates the inventory data structure of an inventory position by indicating an identifier (stock management unit or SKU, etc.) of the inventory item placed at the inventory position. The inventory integration engine also updates the inventory data structure of a shopping store by updating those quantities stocked in the shopping store. The stock position and the store stock data are stored in the stock database 160 together with the customer's stock data (also referred to as a log data structure of stock products or a shopping cart data structure).

The store inventory engine 180 provides the state of the inventory item at the inventory position. However, it is always difficult to determine which inventory item is placed on which part of the shelf. This is important information for shopping store managers and employees. In-stock items can be placed in inventory positions based on the shelves and planograms that identify the positions on the shelves where the in-stock items are planned to be stocked. For example, ketchup bottles may be stocked in a predetermined left portion of all shelves in an inventory display structure that forms a row of arrangements. Over time, customers remove ketchup bottles from the shelves and place them in their respective baskets or shopping carts. Some customers may return ketchup bottles to another part of the same shelf within the same inventory display structure. Customers may also return ketchup bottles to other inventory display shelves in the shopping store. The store realogram engine 190 (hosted on network node 106) produces a realogram that can be used to identify the portion of the shelf on which the ketchup bottles are placed at time "t". This information can be used by the system to generate notifications to employees with misplaced ketchup bottle locations.

This information is also used across inventory items within a real space area to generate a data structure referred to herein as a realogram that keeps track of the location of inventory items within a real space area. obtain. A shopping store generated by a store database engine 190 that reflects the current state of the goods in stock and, in some embodiments, the state of the goods in stock at a specified time "t" over a time interval. The realogram can be stored in the realogram database 170.

The actual communication path via the network 181 to the network node 104 hosting the store inventory engine 170 and to the network node 106 hosting the store realogram engine 190 is on a public network and / or a private network. Can be point-to-point. Communication can be over various networks 181 such as private networks, SOAP, XML circuits, or the Internet, with appropriate application programming interfaces (APIs) and data exchange formats such as REST (Representational State Transfer). ), JSON (JavaScript (trademark) Object Notation), XML (Extensible Markup Language), SOAP (Simple Object Access Protocol), JMS (Java (trademark) Message Service), and / or Java platform module system, etc. be able to. All communications can be encrypted. Communication is generally via protocols such as EDGE, 3G, 4GLTE, Wi-Fi, and WiMAX, LAN (local area network), WAN (wide area network), and telephone network (public exchange telephone network (public exchange telephone network). PSTN)), session initiation protocol (SIP), wireless network, point-to-point network, star network, token ring network, hub network, internet (including mobile internet). In addition, various authorization and authentication techniques such as username / password, OAuth, Kerberos, RSA SecurID, digital certificates, etc. can be used to protect communications.

The techniques disclosed herein are database systems, multi-tenant environments, or database embodiments compatible with Oracle ™, relational database embodiments compatible with IBM DB2 Enterprise Server ™. , A relational database embodiment such as a relational database embodiment compatible with MySQL ™ or PostgreSQL ™ or a relational database embodiment compatible with Microsoft SQL Server ™, or Vampire ™. ), A non-relational database embodiment compatible with Apache Cassandra ™, a non-relational database embodiment compatible with BigTable ™, or HBase ™. ) Or a non-relational database embodiment compatible with DynamoDB ™, etc., which may be implemented in the context of any computer-implemented system including a NoSQL ™ non-relational database embodiment. In addition, the disclosed technology may include various programming models such as MapReduce ™, bulk synchronous programming, MPI primitives, or Apache Storm ™, Apache Spark ™, Apache Kafka ™, Apache Link ™. ), Truviso ™, Amazon Sparksarch Services ™, Amazon Web Services (AWS) ™, IBM Info-Sphere ™, Borealis ™, and Yahoo! It can be performed using various scalable batch and stream management systems such as S4 ™.

[Camera placement]

The camera 114 is arranged to track an articulated subject (or entity) in three-dimensional (abbreviated as 3D) real space. In an exemplary embodiment of a shopping store, the real space can include an area of the shopping store where goods for sale are stacked on shelves. Points in real space can be represented by a (x, y, z) coordinate system. Each point in the real space area to which the system is applied is covered by the field of view of two or more cameras 114.

In a shopping store, shelves and other inventory display structures can be arranged in various ways, such as along the side walls of the shopping store, in rows forming aisles, or in combination of the two configurations. FIG. 2A shows the arrangement of the shelf unit A 202 and the shelf unit B 204 forming the aisle 116a as seen from one end of the aisle 116a. The two cameras, camera A 206 and camera B 208, are located in passage 116a at a predetermined distance from the ceiling 230 and floor 220 of the shopping store on an inventory display structure such as shelves such as shelf unit A 202 and shelf unit B 204. Placed on top. The camera 114 has a field of view that includes each portion of the inventory display structure and the floor area in the real space, and includes a camera arranged on the inventory display structure. As shown in FIG. 2A, the field of view 216 of the camera A 206 and the field of view 218 of the camera B 208 overlap each other. The coordinates in real space of the members of the set of candidate joints identified as the subject identify the position of the subject in the floor area.

In an exemplary embodiment of a shopping store, the real space can include all of the floors 220 in the shopping store. The cameras 114 are arranged and oriented so that the floor 220 and shelf areas are visible by at least two cameras. The camera 114 also covers the floor space in front of the shelves 202 and 204. The angle of the camera is chosen to have both a steep, straight and angled viewpoint, which gives a more complete body image of the customer. In one embodiment, the camera 114 is configured to be at least 8 feet high throughout the shopping store. FIG. 13 shows an explanatory diagram of such an embodiment.

In FIG. 2A, the subject 240 stands beside the shelf unit B 204 of the inventory display structure, and one hand is located near the shelf (invisible) in the shelf unit B 204. FIG. 2B is a perspective view of a shelf unit B 204 having four shelves, shelves 1, shelves 2, shelves 3, and shelves 4 arranged at different heights from the floor. In-stock items are stocked on these shelves.

[3D scene generation]

Positions in real space are represented as points (x, y, z) in the real space coordinate system. The "x" and "y" represent positions on a two-dimensional (2D) plane that can be the floor 220 of the shopping store, and the value "z" is a point on the 2D plane on the floor 220 in one configuration. The height. The system combines 2D images from two or more cameras to generate a three-dimensional position for joints and inventory events (putting goods on shelves and taking goods from shelves) within a real-space area. This section describes the process for generating 3D coordinates of joints and inventory events. The process is also referred to as 3D scene generation.

Two types of camera calibration, namely internal calibration and external calibration, are performed prior to using the system 100 in training or inference mode to track in-stock items. Internal calibration calibrates the internal parameters of the camera 114. Examples of internal camera parameters include focal length, principal point, skew, fisheye factor, and the like. Various techniques for internal camera calibration can be used. One such technique was presented by Zhang in November 2000, Vol. 22, No. 11, "Flexible New Techniques for Camera Calibration," presented in the IEEE Transaction on Pattern Analysis and Machine Intelligence. There is.

In external calibration, external camera parameters are calibrated to generate mapping parameters for converting 2D image data into real space 3D coordinates. In one embodiment, one articulated subject, such as a person, is introduced into real space. The articulated subject moves in real space on a path that passes through the field of view of each camera 114. At any given point in real space, the articulated subject is within the field of view of at least two cameras forming a 3D scene. However, the two cameras have different views of the same 3D scene in their respective two-dimensional (2D) image planes. Features within a 3D scene, such as the left wrist of an articulated subject, are seen by two cameras at different positions in each 2D image plane.

Point correspondence is established among all camera pairs that have overlapping fields of view for a given scene. Since each camera has a different field of view in the same 3D scene, the point correspondence is two pixel positions (one position from each camera with overlapping fields of view) that represent the projection of the same point in the 3D scene. For external calibration, the results of the image recognition engines 112a-112n are used to identify many point correspondences for each 3D scene. The image recognition engine identifies the position of the joint as the (x, y) coordinates of the pixels in the 2D image plane of each camera 114, eg, row and column numbers. In one embodiment, the joint is one of 19 different types of joints in an articulated subject. As the articulated subject moves through the field of view of different cameras, the tracking engine 110 captures each (x, y) coordinates of 19 different types of joints of the articulated subject used for calibration for each image of the camera 114. Receive from.

For example, consider a case where both the image from the camera A and the image from the camera B are taken at the same time point in an overlapping field of view. The image from camera A has pixels that correspond to the pixels of the synchronized image from camera B, and there is a particular point on an object or surface in the field of view of both camera A and camera B, which are both. It is considered that it is captured in the pixels of the image frame of. External camera calibration identifies a number of such points and is referred to as corresponding points. Since there is one articulated subject in the field of view of camera A and camera B during calibration, the major joints of this articulated subject, such as the center of the left wrist, are identified. If these major joints are visible within the image frame from both camera A and camera B, they are assumed to represent corresponding points. This process is repeated for many image frames to build a large set of corresponding points for all camera pairs with overlapping fields of view. In one embodiment, the image is streamed from all cameras at a rate of 30 FPS (frames / second) or higher, in full RGB (red, green, and blue) colors at a resolution of 720 pixels. These images are in the form of a one-dimensional array (also called a flat array).

The large number of images collected above for an articulated subject can be used to determine correspondence points between cameras with overlapping fields of view. Consider two cameras A and B with overlapping fields of view. The plane that passes through the center of the cameras A and B and the joint position (also called the feature point) of the 3D scene is called the "epipolar plane", and the intersection of the epipolar plane and the 2D image planes of the cameras A and B is the "epipolar line". Is defined as. Given these correspondence points, the correspondence points from camera A can be accurately mapped to epipolar lines in the field of view of camera B that are guaranteed to intersect the correspondence points in the image frame of camera B. The conversion is determined. The transformation is generated using the image frames collected above for the articulated subject. It is known in the art that this transformation is non-linear. Further, it is known that in the general form, it is necessary to correct the radial distortion of the lens of each camera as well as the non-linear coordinate transformation moving to and from the projected space. In external camera calibration, the approximation to the ideal nonlinear transformation is determined by solving the nonlinear optimization problem. This non-linear optimization function is performed by the subject tracking engine 110 to identify the same joint in the outputs (array of joint data structures) of various image recognition engines 112a-112n that process images of cameras 114 with overlapping fields of view. used. The results of internal camera calibration and external camera calibration are stored in the calibration database 170.

Various techniques can be used to determine the relative positions of points in the image of the camera 114 in real space. For example, Longuet-Higgins has published "A computer algorithm for reconstructing a scene from two projects" (Nature, Vol. 293, September 10, 1981). In this paper, it is presented to calculate the three-dimensional structure of a scene from a correlation pair of perspective projections when the spatial relationship between the two projections is unknown. The Longuet-Highgins paper presents a method for determining the position of each camera in real space with respect to other cameras. In addition, the technique enables triangulation of articulated subjects in real space and uses images from cameras 114 with overlapping fields of view to identify z-coordinate values (height from the floor). Let any point in real space, for example, the end of a shelf unit in one corner of real space be a (0,0,0) point on the (x, y, z) coordinate system in real space.

In one embodiment of the technique, external calibration parameters are stored in two data structures. The first data structure stores unique parameters. The unique parameter represents a projective transformation from 3D coordinates to 2D image coordinates. The first data structure includes unique parameters for each camera, as shown below. All data values are floating point numbers. This data structure stores a 3x3 eigenmatrix represented as a "K" and a strain coefficient. The strain coefficients include six radial strain coefficients and two tangential strain coefficients. Radial distortion occurs when a ray bends more near the edge of the lens than its optical center. Tangent distortion occurs when the lens and image plane are not parallel. The following data structure shows the values of the first camera only. Similar data is stored for all cameras 114.

{
1: {
K: [[x, x, x], [x, x, x], [x, x, x]],
distortion _coefficients: [x, x, x, x, x, x, x, x]
},
......
}

The second data structure is a 3x3 elementary matrix (F), a 3x3 essential matrix (E), a 3x4 projection matrix (P), a 3x3 rotation matrix (R), and a 3x3 rotation matrix (R) for each camera pair. The 3 × 1 translation vector (t) is stored. This data is used to convert points in the reference frame of one camera to the reference frame of another camera. For each pair of cameras, eight homography coefficients are also stored to map the plane of the floor 220 from one camera to another. The elementary matrix is the relationship between two images in the same scene, constraining where point projections from the scene can occur in both images. The required matrix is also the relationship between two images in the same scene with the camera calibrated. The projection matrix gives a vector space projection from a 3D real space to a subspace. The rotation matrix is used to perform rotations in Euclidean space. The translation vector "t" represents a geometric transformation that moves all points in a figure or space by the same distance in a given direction. The homography floor factor is used to combine images of the features of the subject on the floor 220 seen by cameras with overlapping fields of view. The second data structure is shown below. Similar data is stored for all camera pairs. As mentioned above, x represents a floating point number.

{
1: {
2: {
F: [[x, x, x], [x, x, x], [x, x, x]],
E: [[x, x, x], [x, x, x], [x, x, x]],
P: [[x, x, x, x], [x, x, x, x], [x, x, x, x]],
R: [[x, x, x], [x, x, x], [x, x, x]],
t: [x, x, x],
homography_floor_coefficients: [x, x, x, x, x, x, x, x]
}
},
.......
}

[2D map and 3D map]

The inventory position of a shelf or the like in a shopping store can be identified by a unique identifier (for example, a shelf ID). Similarly, a shopping store can be identified by a unique identifier (eg, store ID). Two-dimensional (2D) and three-dimensional (3D) map databases 140 identify stock locations within real-space areas along their respective coordinates. For example, in a 2D map, positions in the map define a plane formed perpendicular to the floor 220, i.e. a two-dimensional region on the XZ plane, as shown in FIG. The map defines the area of the stock position where the stock goods are placed. In FIG. 3, the 2D view 360 of the shelf 1 in the shelf unit B 204 is formed by four coordinate positions (x1, z1), (x1, z2), (x2, z2), and (x2, z1). An area is indicated and a 2D area in which inventory products are arranged on the shelf 1 is defined. Similar 2D areas are defined for all inventory positions within all shelf units (or other inventory display structures) within a shopping store. This information is stored in the map database 140.

In a 3D map, a position in the map defines a three-dimensional region in 3D real space defined by X, Y, and Z coordinates. The map defines the volume of the stock position where the stock goods are placed. In FIG. 3, the 3D view 350 of the shelf 1 in the shelf unit B 204 has eight coordinate positions (x1, y1, z1), (x1, y1, z2), (x1, y2, z1) that define the 3D region. , (X1, y2, z2), (x2, y1, z1), (x2, y1, z2), (x2, y2, z1), (x2, y2, z2). Is placed within its 3D area on shelf 1. A similar 3D area is defined for inventory locations in all shelf units in the shopping store and is stored in the map database 140 as a 3D map of real space (shopping store). As shown in FIG. 3, the coordinate positions along the three axes can be used to calculate the length, depth, and height of the stock position.

In one embodiment, the map identifies the configuration of a unit of volume that correlates with a portion of the inventory position on the inventory display structure within an area of real space. Each part is defined by a start position and an end position along the three axes of real space. A similar configuration of the inventory position portion can also be generated using a 2D map inventory position that divides the front view of the display structure.

[Joint data structure]

The image recognition engines 112a-112n receive an image sequence from the camera 114 and process the image to generate a corresponding array of joint data structures. The system includes processing logic that tracks the location of multiple subjects (or customers in a shopping store) within an area of real space using an image sequence generated by multiple cameras. In one embodiment, 19 possibilities of the subject in each element of the image that the image recognition engines 112a-112n can use to identify the subject in the area where the inventory may be taken or placed. Identify one of the joints. Possible joints can be divided into two categories: ankle joints and non-ankle joints. The 19th type of joint classification is for all non-joint features of the subject (ie, elements of the image that are not classified as joints). In other embodiments, the image recognition engine may be configured to specifically identify the position of the hand. Also, for the purpose of identifying the subject and linking the detected position of the subject's hand to the subject as the subject moves through the store, other techniques such as user check-in procedures or biometric identification processing. Can be deployed.
Ankle joint:
Ankle joint (left and right)
Non-ankle:
neck
nose
Eyes (left and right)
Ears (left and right)
Shoulder (left and right)
Elbow (left and right)
Wrist (left and right)
Buttocks (left and right)
Knee (left and right)
Non-joint

An array of joint data structures for a particular image classifies the elements of a particular image by joint type, time of the particular image, and coordinates of the elements within the particular image. In one embodiment, the image recognition engines 112a-112n are convolutional neural networks (CNNs), the joint type is one of 19 types of joints in the subject, and the time of a particular image is determined by the source camera 114 for a particular image. It is a time stamp of the generated image, and the coordinates (x, y) specify the position of the element on the 2D image plane.

The output of the CNN is a matrix of confidence arrays for each image per camera. The matrix of confidence arrays is transformed into an array of joint data structures. The joint data structure 400 as shown in FIG. 4 is used to store information about each joint. The joint data structure 600 identifies the x and y positions of an element in a particular image within the 2D image space of the camera from which the image is received. The joint number identifies the type of joint identified. For example, in one embodiment, the value ranges from 1 to 19. A value of 1 indicates that the joint is the left ankle, a value of 2 indicates that the joint is the right ankle, and so on. The type of joint is selected using the confidence array for that element in the CNN's output matrix. For example, in one embodiment, if the value corresponding to the left ankle joint is the highest in the confidence array of the image element, the value of the joint number is "1".

The confidence frequency indicates the degree of confidence of the CNN in predicting the joint. If the confidence value is high, the CNN is confident in its expectations. An integer ID is assigned to the joint data structure to uniquely identify the joint data structure. Following the above mapping, the output matrix 540 of the reliability array for each image is converted into an array of joint data structures for each image. In one embodiment, joint analysis comprises performing k-nearest neighbors, Gaussian mixing, and various image morphological transformation combinations for each input image. This result contains an array of joint data structures that can be stored in bitmask format in a ring buffer that maps the number of images to a bitmask at each point in time.

[Subject tracking engine]

［被写体データ構造］ The tracking engine 110 is configured to receive an array of joint data structures generated by the image recognition engines 112a-112n that correspond to the images in the image sequence from cameras with overlapping fields of view. The array of joint data structures per image is sent by the image recognition engines 112a-112n to the tracking engine 110 via the network 181. The tracking engine 110 transforms the coordinates of the elements in the array of joint data structures corresponding to the various image sequences into candidate joints with coordinates in real space. Positions in real space are covered by the field of view of two or more cameras. The tracking engine 110 includes logic for identifying a set of candidate joints having coordinates (joint constellations) in real space as subjects in real space. In one embodiment, the tracking engine 110 is used to accumulate an array of joint data structures from the image recognition engine and identify candidate joint constellations for all cameras at a given time point. This information is stored in the subject database 140 as a dictionary. The dictionary can be organized in the form of key-value pairs, where the key is the camera ID and the values are an array of joint data structures from the camera. In such embodiments, this dictionary is used in heuristics-based analysis to determine candidate joints and assign joints to subjects. In such embodiments, the high level inputs, processes, and outputs of the tracking engine 110 are shown in Table 1. Details of the logic applied by the subject tracking engine 110, which combines candidate joints to generate a subject and track the movement of the subject in a real-space area, are described in US Pat. No. 10,055, Issued August 21, 2018. No. 853, "Recognizing and Tracking Subjects Using an Image Recognition Engine", which is incorporated herein by reference.

Table 1: Inputs, processes, and outputs from the subject tracking engine 110 in an exemplary embodiment.

[Subject data structure]

The subject tracking engine 110 uses heuristics to connect the joints of the subject identified by the image recognition engines 112a-112. At that time, the subject tracking engine 110 creates a new subject and updates the position of the existing subject by updating the joint positions of the new subjects. The subject tracking engine 110 uses a triangulation technique to project joint positions from 2D spatial coordinates (x, y) to 3D real space coordinates (x, y, z). FIG. 5 shows a subject data structure 500 for storing a subject. The data structure 500 stores subject-related data as a key value dictionary. The key is the frame ID and the value is another key-value dictionary, where the key is the camera ID and the value is a list of 18 joints (of the subject) and their position in real space. The subject data is stored in the subject database. For each new subject, a unique identifier used to access the subject's data in the subject database is also assigned.

In one embodiment, the system identifies the joints of the subject and creates the skeleton of the subject. The skeleton is projected onto the real space and indicates the position and orientation of the subject in the real space. This is also called "posture estimation" in the field of machine vision. In one embodiment, the system displays the orientation and position of a subject in real space on a graphical user interface (GUI). In one embodiment, the subject identification and image analysis are anonymous, that is, the unique identifier assigned to the subject created by the joint analysis does not identify the subject's personal identification information, as described above.

In this embodiment, the constellation of the identified joints of the subject, generated by time series analysis of the joint data structure, can be used to locate the hand of the subject. For example, the position of the wrist joint alone or the position based on the projection of the combination of the wrist joint and the elbow joint can be used to identify the position of the hand of the identified subject.

[Inventory event]

FIG. 6 shows a subject 240 for taking out an in-stock item from the shelf of the shelf unit B 204 in the top view 610 of the passage 116a. The disclosed technique uses an image sequence generated by at least two cameras within multiple cameras to locate inventory events. The joints of a single subject can appear within the image frames of multiple cameras within each image channel. In the example of a shopping store, the subject moves in an area of real space, takes out the goods from the stock position, and puts the goods back in the stock position. In one embodiment, the system uses a pipeline of convolutional neural networks called WhatCNN and WhenCNN to predict inventory events (putting or taking, also called plus or minus events).

A dataset containing the subject identified by the joints in the subject data structure and the corresponding image frames from the sequence of image frames per camera is given as input to the bounded box generator. The bounded box generator implements logic that processes the dataset to specify a bounded box that contains the image of the identified subject's hand in the image in the image sequence. The bounded box generator uses, for example, the position of the wrist joint (for each hand) and the elbow joint in the articulated data structure 500 corresponding to each source image frame in each source image frame for each camera. Identify the position of the hand. In one embodiment where the coordinates of the joints in the subject data structure indicate the position of the joints in 3D real space coordinates, the bounded box generator sets the joint positions from the 3D real space coordinates to 2D in the image frame of each source image. Map to coordinates.

The bounded box generator creates a bounded box for each hand in the image frame in the circular buffer for each camera 114. In one embodiment, the bounded box is a 128 pixel (width) x 128 pixel (height) portion of the image frame, and the hand is located in the center of the bounded box. In another embodiment, the size of the bounded box is 64 pixels x 64 pixels or 32 pixels x 32 pixels. For m subjects in the image frame from the camera, there can be up to 2m hands, and thus 2m bounded boxes. However, in practice, less than 2 m of hands are visible in the image frame due to shielding by other subjects or other objects. In one exemplary embodiment, the position of the subject's hand is inferred from the positions of the elbow and wrist joints. For example, the position of the right hand of the subject is extrapolated as extrapolation amount × (p2-p1) + p2 using the position of the right elbow (identified as p1) and the position of the right wrist (identified as p2). .. Here, the extrapolation amount is 0.4. In another embodiment, the joints CNN112a-112n are trained using left and right hand images. Therefore, in such an embodiment, the joints CNN112a-112n directly identify the position of the hand within the image frame per camera. The hand position per image frame is used by the bounded box generator to generate the identified bounded box per hand.

WhatCNN is a convolutional neural network trained to process a specified bounded box in an image to generate a hand classification for an identified subject. One trained WhatCNN processes image frames from one camera. In an exemplary embodiment of a shopping store, for each hand in each image frame, WhatCNN identifies whether the hand is empty. WhatCNN also has a SKU (stock keeping unit) number for the goods in stock in the hand, a non-SKU product (ie, does not belong to the shopping store stock) with a confidence value indicating the goods in the hand, and a hand in the image frame. Identifies the status of the location of.

The output of the WhatCNN model of all cameras 114 is processed by a single IfCNN model for a given time period. In the example of a shopping store, WhenCNN performs a time series analysis on both hands of the subject to identify whether the subject takes the store inventory from the shelf or puts the store inventory on the shelf. The disclosed technique uses an image sequence generated by at least two of the cameras to locate the inventory event. WhenCNN performs analysis of the dataset from image sequences from at least two cameras to determine the location of the inventory event in three dimensions and identify the goods associated with the inventory event. A time series analysis of the output of WenCNN per subject over a period of time is performed to identify inventory events and their time of occurrence. A non-maximum suppression (NMS) algorithm is used for this purpose. If one inventory event (ie, placing or taking goods by the subject) is detected multiple times by WenCNN (from both the same camera and multiple cameras), the NMS removes the extra event for the subject. NMS rescoring includes two major tasks: "matching loss", which penalizes extra detection, and neighboring "joint processing" to see if better detection is at hand. Ring technology.

The true events of taking and placing for each subject are further processed by calculating the average SKU logit for the 30 image frames before the image frame having the true event. Finally, the maximum value argument (abbreviated as arg max or arg max) is used to determine the maximum value. Inventory items classified by argmax value are used to identify inventory items placed on or taken from shelves. The disclosed technology assigns inventory events to a subject by assigning inventory-related inventory events to the subject's log data structure (or shopping cart data structure). Inventory items are added to the SKU (also known as shopping cart or basket) log for each subject. The image frame identifier "frame ID" of the image frame that led to the detection of the inventory event is also stored with the identified SKU. The logic for assigning an inventory event to a subject matches the position of the inventory event with the position of one of the customers among multiple customers. For example, the image frame can be used to identify the 3D position of an inventory event represented by the position of the subject's hand at at least one time point in the sequence classified as an inventory event using the subject data structure 500. , And can be used to determine the inventory position from where the goods were taken out or placed. The disclosed technique uses image sequences generated by at least two of the cameras to locate inventory events and create inventory event data structures. In one embodiment, the inventory event data structure stores a product identifier, an indicator to place or take, three-dimensional coordinates of an area in real space, and a time stamp. In one embodiment, the inventory event is stored in the inventory event database 150.

The location of an inventory event (placement and picking of inventory items by a subject within an area of space) is a store planogram to identify the inventory position, such as a shelf on which the subject has taken out or placed the item. Or you can compare it with other maps. Example 660 shows the determination of a shelf in a shelf unit by calculating the shortest distance from the hand position 640 associated with the inventory event. This shelf determination is then used to update the shelf inventory data structure. FIG. 7 shows an exemplary inventory data structure 700 (also referred to as a log data structure). This inventory data structure stores the inventory of a subject, a shelf, or a store as a key value dictionary. The key is the unique identifier of the subject, shelf or store, the value is another key-value dictionary, in this case the key is the product identifier, such as a stock keeping unit (SKU), and the value is the inventory event forecast. It is a number that identifies the quantity of the product together with the "frame ID" of the brought image frame. The frame identifier (“frame ID”) can be used to identify the image frame that resulted in the identification of the inventory event that resulted in the association between the inventory item and the subject, shelf, or store. In another embodiment, the "camera ID" that identifies the source camera can be combined with the frame ID and stored in the inventory data structure 700. In one embodiment, the "frame ID" is the subject identifier because the frame holds the subject's hand in the bounded box. In another embodiment, another type of identifier, such as a "subject ID" that explicitly identifies a subject in a real space area, can be used to identify the subject.

When the shelf inventory data structure is integrated with the subject's log data structure, the shelf inventory is reduced to reflect the quantity of goods that the customer has taken out of the shelf. If the customer puts the goods on the shelves or the employee stocks the goods on the shelves, the goods are added to the inventory data structure of each stock position. Over time, this process results in an update of the shelf inventory data structure for all inventory locations within the shopping store. The inventory data structure of the inventory position in the real space area is integrated and the inventory data structure of the real space area showing the total number of products of each SKU in the store at that time is updated. In one embodiment, such an update is performed after each inventory event. In another embodiment, the store inventory data structure is updated periodically.

For more information on the WatCNN and ThenCNN embodiments of detecting inventory events, see US Patent Application No. 15 / 907,112, filed February 27, 2018, "Detection of Placing and Taking Goods Using Image Recognition." Shown in, which is incorporated herein by reference, as if fully described herein.

[Real-time shelf and store inventory update]

FIG. 8 is a flowchart showing a processing step for updating the shelf inventory structure in the real space area. The process begins at step 802. In step 804, the system detects an event to be taken or placed within an area of real space. Inventory events are stored in the inventory event database 150. The inventory event record includes a product identifier such as a SKU, a time stamp, and the location of the event in a real space 3D area indicating a position along the 3D x, y, and z. Inventory events also include a place or take indicator to identify whether the subject has placed the item on the shelf (also known as a plus inventory event) or removed the item from the shelf (also known as a minus inventory event). The inventory event information is combined with the output from the subject tracking engine 110 to identify the subject associated with this inventory event. The results of this analysis are then used to update the subject's log data structure (also known as the shopping cart data structure) in the inventory database 160. In one embodiment, the subject identifier (eg, "subject ID") is stored in the inventory event data structure.

The system can use the subject's hand position (step 806) associated with the inventory event to find the position of the closest shelf in the inventory display structure (also referred to as the shelf unit above) in step 808. The store inventory engine 180 calculates the distance of the hand to a two-dimensional (2D) area or area on the xz plane (perpendicular to the floor 220) of the inventory position in the shopping store. The 2D area of the inventory position is stored in the map database 140 of the shopping store. Suppose the hand is represented by a _{point E (x event} , y _event , z _{event) in real space.} The shortest distance D from the point E in the real space to any point P on the plane can be determined by projecting the vector PE onto the normal vector n with respect to the plane. Existing mathematical techniques can be used to calculate the distance of the hand to all planes representing the 2D region of the stock position.

In one embodiment, the disclosed technique comprises calculating the distance from the position of the inventory event to the inventory position on the inventory display structure, and matching the inventory event with the inventory position based on the calculated distance. Match the position of the inventory event with the inventory position by performing the procedure. For example, the inventory position (shelf, etc.) that is the shortest distance from the position of the inventory event is selected, and the inventory data structure of this shelf is updated in step 810. In one embodiment, the position of the inventory event is determined by the position of the subject's hand along three coordinates in real space. If the inventory event is a take event (or negative event) indicating that the ketchup bottle was taken by the subject, the inventory on the shelf is updated by reducing the number of ketchup bottles by one. Similarly, if the inventory event is a placing event indicating that the subject places a bottle of ketchup on the shelf, the number of ketchup bottles is increased by one to update the inventory on the shelf. Similarly, the store inventory data structure is updated accordingly. The quantity of goods placed in the inventory position is incremented by the same number in the store inventory data structure. Similarly, the quantity of goods taken from the inventory position is deducted from the inventory data structure of the store in the inventory database 160.

At step 812, it is checked whether the planogram is available to the shopping store or whether it can be known that the planogram is available. A planogram is a data structure that maps inventory items to inventory locations within a shopping store, which can be based on a plan for the placement of inventory items within a store. If a planogram is available for the shopping store, in step 814 the goods placed on the shelves by the subject are compared to the goods on the shelves in the planogram. In one embodiment, the disclosed technique includes logic for determining misplaced goods when an inventory event matches an inventory position that does not match the planogram. For example, if the SKU of the goods associated with the inventory event matches the placement of the goods in stock at the stock position, the position of the goods is correct (step 816), otherwise the goods are misplaced. In one embodiment, in step 818, a notification is sent to the employee to remove the misplaced merchandise from the current storage position (such as a shelf) and move it to its correct inventory position according to the planogram. In step 820, the system checks whether the subject leaves the shopping store using the speed, orientation, and proximity to the exit of the store. If the subject is not about to leave the store (step 820), processing resumes at step 804. Otherwise, if it is determined that the subject is leaving the store, step 822 integrates the subject's log data structure (or shopping cart data structure) and the store's inventory data structure.

In one embodiment, if in step 810 the goods in the subject's shopping cart data structure are not deducted from the store inventory, the integration comprises deducting these goods from the store inventory data structure. In this step, the system can also identify items in the shopping cart data structure of the subject with a low identification confidence score and send a notification to a store employee located near the store exit. The employee can then see the item with the low identification confidence score in the customer's shopping cart. This process does not require the store employee to compare all the products in the customer's shopping cart with the customer's shopping cart data structure, and only the products with a low confidence score are sent to the store employee by the system. Identified and confirmed by store employees. The process ends in step 824.

[Architecture for real-time shelf and store inventory updates]

An example architecture of a system in which customer inventory, inventory location (eg, shelves) inventory, and store inventory (eg, entire store) data structures are updated using placing and taking goods by customers in a shopping store. It is shown in FIG. 9A. Since FIG. 9A is an architectural diagram, certain details are omitted to improve clarity of description. The system shown in FIG. 9A receives image frames from a plurality of cameras 114. As mentioned above, in one embodiment, the cameras 114 can be temporally synchronized with each other so that the images are acquired simultaneously or close in time and at the same image capture rate. Images acquired by all cameras covering areas of real space that are simultaneous or close in time represent different scenes of the synchronized image at a given point in time with a subject having a fixed position in real space. Synchronized in the sense that it can be identified in the processing engine. The image is stored in the circular buffer 902 of the image frame for each camera.

A "subject identification" subsystem 904 (also referred to as a first image processor) processes an image frame received from the camera 114 to identify and track a subject in real space. The first image processor includes a subject image recognition engine that detects joints of a subject in real space. The joints are connected to form a subject and are tracked as movement in real space. The subject is anonymous and is tracked using the internal identifier "subject ID".

The "Region Proposal" subsystem 908 (also referred to as a third image processor) includes a foreground image recognition engine that receives corresponding image sequences from multiple cameras 114 and is a semantically important object in the foreground (ie, ie. The object is recognized when the customer, the customer's hand, and the inventory item) are associated with placing and taking the inventory item over time in the image from each camera. The region proposal subsystem 908 also receives the output of the subject identification subsystem 904. The third image processor processes the image sequence from the camera 114 to identify and classify the foreground changes represented by the images in the corresponding image sequence. The third image processor processes the identified foreground change to detect that the identified subject takes inventory and places the inventory on the inventory display structure by the identified subject. Create a detection set for. In one embodiment, the third image processor comprises a convolutional neural network (CNN) model such as WhatCNN described above. The first detection set is also referred to as foreground detection of placing and taking inventories. In this embodiment, the output of WhatCNN is processed by a second convolutional neural network (WhenCNN) to place an inventory item in the inventory position, and on the inventory position in the inventory display structure by customers and store employees. Create a first set of detections that identifies the events taken by the in-stock items. Details of the domain proposal subsystem are set forth in US Patent Application No. 15 / 907,112, filed February 27, 2018, "Detection of Placing and Taking Goods Using Image Recognition". Is incorporated herein by reference as if fully described herein.

In another embodiment, the architecture detects placing and taking inventory items and uses them in parallel with a third image processor to associate these placing and taking with the subject in the shopping store. Includes a "semantic difference extraction" subsystem (also known as a second image processor) that can. This semantic difference extraction subsystem includes a background image recognition engine, which receives corresponding image sequences from multiple cameras, eg, in the background (ie, a shelf-like inventory display structure). Recognize such differences as the semantically significant differences relate to placing and taking stock items over time in the images from each camera. The second image processor receives the output of the subject identification subsystem 904 and the image frame from the camera 114 as inputs. For more information on the "Semantic Difference Extraction" subsystem, see US Pat. No. 10,127,438, filed April 4, 2018, "Prediction of Inventory Events Using Semantic Difference Extraction," and April 2018. It is set forth in U.S. Patent Application No. 15 / 945,473, "Prediction of Inventory Events Using Foreground / Background Processing," filed on the 4th, both of which appear to be fully described herein. Incorporated herein by reference. The second image processor processes the identified background change to detect that the identified subject takes inventory and places the inventory on the inventory display structure by the identified subject. Create a detection set for. The second detection set is also referred to as background detection for placing and taking inventories. In the example of a shopping store, the second detection identifies an inventory item taken from an inventory position or an inventory item placed on the inventory position by a customer or store employee. The semantic difference extraction subsystem contains logic that associates the identified background changes with the identified subject.

The system described in FIG. 9A includes selection logic for processing the first and second detection sets to generate a log data structure containing a list of in-stock items for the identified subject. For placing and taking in real space, the selection logic selects the output from either the semantic difference extraction subsystem or the region suggestion subsystem 908. In one embodiment, the selection logic uses a confidence score generated by the semantic difference extraction subsystem for the first detection set and a confidence score generated by the region proposal subsystem for the second detection set. And make a selection. Log data structure 700 (also known as a shopping cart data structure) that contains a list of in-stock items and their quantities associated with the identified subject, with the output of the subsystem selected to have a higher confidence score for a particular detection. Used to generate. The shelf and store inventory data structures are updated using the subject log data structure as described above.

The subject exit detection engine 910 determines whether the customer is moving toward the exit door and transmits a signal to the store inventory engine 190. The store inventory engine determines if one or more goods in the customer's log data structure 700 have the low discriminant confidence score determined by the second or third image processor. If so, the inventory integration engine sends a notification to a store employee located near the exit to confirm the goods purchased by the customer. The subject, the inventory position, and the inventory data structure of the shopping store are stored in the inventory database 160.

FIG. 9B is a system in which customer inventory, inventory location (eg, shelves) inventory, and store inventory (eg, entire store) data structures are updated using placing and taking goods by customers in a shopping store. Shows another architecture of. Since FIG. 9A is an architectural diagram, certain details have been omitted to improve clarity of description. As mentioned above, the system receives image frames from a plurality of synchronized cameras 114. WhatCNN 914 uses an image recognition engine to determine the goods in the hands of a customer in an area of real space (shopping store, etc.). In one embodiment, there is one WhatCNN for each camera 114, which performs image processing on a sequence of image frames generated by each camera. WhenCNN912 performs a time series analysis of the output of WhatCNN to identify the event to put or take. Inventory events are stored in database 918 along with product and hand information. This information is then combined with the customer information generated by the customer tracking engine 110 (also referred to herein as the subject tracking engine 110) by the personal and product attribute component 920. The customer log data structure 700 in the shopping store is generated by linking the customer information stored in the database 918.

The disclosed technology uses a sequence of images generated by multiple cameras to detect a customer leaving an area of real space. In response to the detection of customer withdrawal, the disclosed technology updates store inventory in memory for goods related to inventory events caused by the customer. When the exit detection engine 910 detects the departure of customer "C" from the shopping store, the goods purchased by customer "C" are integrated with the store inventory data structure 924 as shown in log data structure 922. , Generate an updated store inventory data structure 926. For example, as shown in FIG. 9B, the customer has purchased two merchandise 1, four merchandise 3 and one merchandise 4. The quantity of each merchandise purchased by customer "C" as shown in the customer's log data structure 922 is deducted from the store inventory 924, the quantity of merchandise 1 is reduced from 48 to 46, and so on. Generates an updated store inventory data structure 926 indicating that the quantities of 3 and 4 have been reduced by the respective quantities of goods 3 and 4 purchased by customer "C". Since the customer "C" did not purchase the product 2, in the updated store inventory data structure 926, the quantity of the product 2 remains the same as the previous one in the current store inventory data structure 924.

In one embodiment, the customer withdrawal detection also initiates an update of the inventory data structure of the inventory position (such as a shopping store shelf) from which the customer has taken out the goods. In such an embodiment, the inventory data structure of the inventory position is not updated immediately after the inventory event to be taken or placed, as described above. In this embodiment, when the system detects a customer leaving, the inventory event associated with the customer is traversed and the inventory event is linked to each inventory location in the shopping store. The inventory data structure of the inventory position determined by this process is updated. For example, if the customer takes two goods 1 from the stock position 27, the stock data structure of the stock position 27 is updated by reducing the quantity of the goods 1 by two. Note that in-stock items can be stocked in multiple inventory locations within a shopping store. The system identifies the inventory position corresponding to the inventory event and therefore updates the inventory position from which the goods are retrieved.

[Store realogram]

The location of in-stock items across the real space in the store, including the inventory position in the shopping store, allows the customer to remove the item from the inventory position and return the item that the customer does not want to purchase to the same position on the same shelf from which the item was taken out. It changes over a period of time when the goods are taken out, returned to different locations on the same shelf, or placed on different shelves. The disclosed technology uses image sequences generated by at least two cameras in multiple cameras to identify inventory events and respond to inventory events to position inventory items within a real-space area. Chase. In some embodiments, the goods in the shopping store are placed according to a planogram that identifies the inventory location (such as a shelf) where the particular goods are planned to be placed. For example, as shown in Example 910 of FIG. 10, the left half of the shelves 3 and 4 is designated as a commodity (stocked in the form of a can). At the beginning of the day or at another inventory tracking interval (identified by time t = 0), inventory positions are considered to be stocked according to the planogram.

The disclosed technology can calculate a "realogram" of a shopping store at any time "t" which is a real-time map of the location of inventories within an area of real space, which is several implementations. In the form, it can be further correlated with the inventory position in the store. Realograms can be used to create planograms by identifying inventory items and locations within the store and mapping them to inventory locations. In one embodiment, the system or method can create a dataset that defines multiple cells having coordinates within an area of real space. The system or method can use the cell length along the coordinates of the real space as an input parameter to divide the real space into datasets that define multiple cells. In one embodiment, the cells are represented as a two-dimensional grid with coordinates within an area of real space. For example, the cell can correlate with a 2D grid (eg, at 1 foot intervals) of the front view of the stock position in the shelf unit (also called the stock display structure), as shown in Illustrative 960 of FIG. As shown in FIG. 10, each grid is defined by its start position and end position on the coordinates of a two-dimensional plane such as x-coordinates and z-coordinates. This information is stored in the map database 140.

In another embodiment, the cells are represented as a three-dimensional (3D) grid with coordinates within an area of real space. In one example, the cell can correlate with the volume on the stock position (or part of the stock position) of the shelf unit in the shopping store, as shown in FIG. 11A. In this embodiment, a map of real space identifies a configuration of units of capacity that can correlate with a portion of inventory position on the inventory display structure within an area of real space. This information is stored in the map database 140. The store realogram engine 190 uses the inventory event database 150 to calculate the realogram of the shopping store at time "t" and stores it in the realogram database 170. The shopping store realogram uses inventory event timestamps stored in the inventory event database 150 to associate inventory items with inventory events matched to cells by their position at any time t. Is shown. The inventory event includes a product identifier, an indicator to place or take, the position of the inventory event represented by the position along the three axes of the real space area, and the time stamp.

The illustration in FIG. 11A shows that at the beginning of the first day at t = 0, the left portion of the stock position in the first shelf unit (forming the columnar array) houses the "ketchup" bottle. The columns of cells (or grids) are shown in black in the schematic visualization, and the cells can be rendered in other colors such as dark green. All other cells remain blank and are not filled with a color that indicates that they do not contain the item. In one embodiment, the visualization of a product in a cell within a realogram is generated for one product at a time indicating its location in the store (in the cell). In another embodiment, the rearogram uses different colors to distinguish and display the position of the set of goods on the stock position. In such an embodiment, a cell can have a plurality of colors corresponding to some goods associated with an inventory event that matches the cell. In another embodiment, other schematic or text-based visualizations are used to indicate inventory items in a cell, such as by enumerating SKUs or names in the cell.

The system uses each count of inventory events to calculate a SKU score (also called a score) at the time of scoring for an inventory item that has a position that matches a particular cell. The calculation of the score for a cell uses the sum of taking and placing inventories weighted by the separation between the time stamp of taking and placing and the time of scoring. In one embodiment, the score is a weighted average of inventory events per SKU. In other embodiments, various scoring calculations can be used, such as counting inventory events per SKU. In one embodiment, the system displays the realogram as a cell within a plurality of cells and an image representing the score of the cell. For example, consider scoring t = 1 (for example, one day later) as an example in FIG. 11A. The time t = 1 realogram represents the score of the "ketchup" product in different shades of black. In the store realogram at time t = 1, all four rows of the first shelf unit and the second shelf unit (behind the first shelf unit) contain "ketchup" products. Cells with high SKU scores in "ketchup" bottles are rendered in darker gray than cells with low scores in "ketchup" bottles. Cells with a ketchup score of zero are not left blank and are not filled with color. Therefore, after time t = 1 (eg, one day later), the realogram presents real-time information about the location of the ketchup bottle on the inventory location in the shopping store. The frequency of generating the realogram can be set according to the request by the management of the shopping store. Realograms can also be generated on demand by store management. In one embodiment, the merchandise location information generated by the realogram is compared to the store planogram to identify misplaced merchandise. Notifications can be sent to store employees who can return misplaced inventory items to a designated inventory position, as shown in the store's planogram.

In one embodiment, the system renders a cell within a plurality of cells and a display image representing the score of the cell. FIG. 11B shows a computing device in which the realogram of FIG. 11A is rendered on the user interface display 1102. Realograms can be displayed on other types of computing devices such as tablets and mobile computing devices. The system can use the color change in the display image representing the cell to indicate the score of the cell. For example, in FIG. 11A, the row of cells accommodating "ketchup" at t = 0 can be represented by the dark green cells in that row. At t = 1, "ketchup" bottles are distributed across the first column of cells into a plurality of cells. The system can represent these cells by using different shades of green to indicate the scores of the cells. Dark green tones indicate higher scores and light green cells indicate lower scores. The user interface displays other generated information and provides tools for invoking and displaying features.

[Calculation of store realogram]

FIG. 12 is a flow chart presenting a processing step for calculating a shelf realogram within a real space area at time t, which may be adapted to other types of inventory display structures. The process starts at step 1202. At step 1204, the system retrieves inventory events in the real space area from the inventory event database 150. The inventory event record includes the goods identifier, the indicator to put or take, the location of the inventory event represented by the position in the three dimensions (x, y, and z, etc.) of the real space area, and the time stamp. The put or take indicator identifies whether the customer (also known as the subject) puts the item on the shelf or removes the item from the shelf. The event to put is also called a plus inventory event, and the event to take is also called a minus inventory event. In step 1206, the inventory event is combined with the output from the subject tracking engine 110 to identify the subject's hand associated with this inventory event.

The system uses the position of the subject's hand (step 1206) associated with the inventory event to determine the position. In some embodiments, the inventory event can be matched in step 1208 with the nearest shelf or other possible inventory position in the shelf unit or inventory display structure. Processing step 808 in the flowchart of FIG. 8 shows details of a technique that can be used to determine the position on the shelf closest to the position of the hand. As described in the procedure in step 808, the shortest distance D from point E in real space to any point P on the plane (representing the front region of the shelf on the xz plane) is the vector PE. It can be determined by projecting onto the normal vector n with respect to the plane. The intersection of the vector PEs with respect to the plane gives the hand the closest point on the shelf. The location of this point is stored in the "point cloud" data structure (step 1210) as a tuple containing the 3D location of the point in the real space area, the SKU of the product, and the time stamp, the latter two being inventory events. -Obtained from the record. If there are more inventory event records (step 1211) in the inventory event database 150, processing steps 1204-1210 are repeated. If it does not exist further, the process proceeds to step 1214.

The disclosed technique includes a data set stored in memory that defines a plurality of cells having coordinates within an area of real space. A cell defines an area of real space bounded by a start position and an end position along an axis. The real space area contains a plurality of inventory positions, and the coordinates of the cells in the plurality of cells can be correlated with the inventory positions in the plurality of inventory positions. The disclosed technique matches the position of the inventory item associated with the inventory event with the cell coordinates and maintains data representing the inventory item matched with the cells in the cells. In one embodiment, the system calculates the distance from the location of the inventory event to the cell in the dataset and a procedure for matching the inventory event with the cell based on the calculated distance (step 808 in the flowchart of FIG. 8). By executing (as described in), the closest cell in the dataset is determined based on the location of the inventory event. This matching of the event location to the nearest cell gives the location of the point cloud data that identifies the cell in which the point cloud data resides (step 1212). In one embodiment, the cell can be mapped to a portion of the inventory position (shelf, etc.) in the inventory display structure. Therefore, by using this mapping, the parts of the shelves are also identified. As mentioned above, cells can be represented as a 2D or 3D grid of real-space areas. The system includes logic to calculate the score at the time of scoring for inventory items that have positions that match a particular cell. In one embodiment, the score is based on a count of inventory events. In this embodiment, the cell scoring uses the sum of taking and placing inventories weighted by the separation between the time stamp of taking and placing and the time of scoring. For example, the score can be a weighted moving average per SKU (also known as a SKU score) and is calculated cell by cell using the "point cloud" data points mapped to the cell:

The SKU score calculated by equation (1) is the sum of the scores of the data points of all the point clouds of the SKU in the cell, where each data point is the number of days from the time stamp of the event to put and take. Weighted by time point_t. Suppose there are two point cloud data points for a "ketchup" product in the grid. The first data point has a time stamp indicating that this inventory event occurred two days before the time "t" at which the realogram was calculated. Therefore, the value of point_t becomes "2". The second data point corresponds to an inventory event that occurred one day before time "t", so point_t is "1". The cell ketchup score (identified by the cell ID that maps to the shelf identified by the shelf ID) is calculated as follows:

As the data points in the point cloud corresponding to the inventory event become older (ie, more days have passed since the event), their contribution to the SKU score diminishes. In step 1216, the top "N" SKUs are selected for the cell with the highest SKU score. In one embodiment, the system includes logic to select a set of inventories per cell based on a score. For example, the value of "N" can be selected as 10 to select the top 10 products per cell based on the SKU score. In this embodiment, the realogram stores the top 10 products per cell. The updated realogram at time t is stored in the realogram database 170 showing the top "N" SKUs per cell in the shelf at time t in step 1218. The process ends in step 1220.

In another embodiment, the disclosed technique does not use a 2D or 3D map of the shelf portion stored in the map database 140 to calculate the point cloud data in the shelf portion corresponding to the inventory event. .. In this embodiment, the 3D real space representing the shopping store is divided into cells represented as 3D cubes (eg, 1 foot cubes). The 3D hand position is mapped to the cell (using each position along the three axes). The SKU score for all products is calculated cell by cell using the above formula (1). The resulting realogram shows the goods in a cell in real space that represents the store, without requiring the location of the shelves in the store. In this embodiment, the data point in the point cloud may be at the same position of the hand corresponding to the inventory event on coordinates in real space, or is close to or close to the position of the hand. It may be at the position of the cell in the area that includes the position. This is because there may not be a map of the shelves, so the hand position is not mapped to the nearest shelf. Therefore, the data points of the point cloud in this embodiment do not necessarily have to be on the same plane. All point cloud data points within a unit of volume in real space (eg, 1 cubic foot) are included in the SKU score calculation.

In some embodiments, the realogram is iteratively calculated and used for time analysis of in-store activity, or an animation to show the movement of inventories in the store over time ( Can be used to generate (like stop motion animation).

[Application example of store realogram]

Store realograms can be used in many operations of shopping stores. Some application examples of realograms are shown in the following paragraphs.

[Restock of in-stock items]

FIG. 13A presents one such application of a store realogram for determining whether an inventory item needs to be restocked in an inventory location (such as a shelf). The process starts at step 1302. At step 1304, the system searches the realogram database 170 for the realogram at scoring "t". In one example, this is a recently generated realogram. The SKU scores of all cells in the rearogram are compared to the threshold scores in step 1306. If the SKU score is above the threshold (step 1308), the process repeats steps 1304 and 1306 for the next inventory item "i". In embodiments that include a planogram, or where a planogram is available, the SKU score of the product "i" is compared to a threshold for cells that match the placement of the stock product "i" in the planogram. In another embodiment, the SKU score of an in-stock item is calculated by filtering the "put" inventory event. In this embodiment, the SKU score reflects the "take" event of the stock item "i" per cell in the realogram that can be compared to the threshold. In another embodiment, the count of "take" inventory events per cell can be used as a score to compare with a threshold for determining restocking of inventory item "i". In this embodiment, the threshold is the minimum count of inventories that need to be stocked at the inventory position.

If the SKU score for inventory item "i" is below the threshold, a warning notice is sent to the store manager or other designated employee indicating that inventory item "i" needs to be restocked (step 1310). ). The system can also identify inventory positions where inventory items need to be restocked by matching cells with SKU scores below the threshold to inventory positions. In another embodiment, the system checks the inventory level of inventory item "i" in the stock room of a shopping store to determine if inventory item "i" needs to be ordered from a wholesaler. Can be done. The process ends in step 1312. FIG. 13B presents an exemplary user interface that displays restock warning notices for in-stock items. Warning notifications can be displayed on the user interface of other types of devices such as tablets and mobile computing devices. Alerts are sent to designated recipients via email, SMS (Short Message Service) on mobile phones, or notifications to store applications installed on mobile computing devices. You can also do it.

[Misplaced inventory items]

In embodiments that include a planogram, or if the store's planogram is available in other ways, for the compliance of the planogram, the rearogram will identify the misplaced merchandise. Is compared with. In such an embodiment, the system includes a planogram that specifies the placement of inventory goods at inventory locations within an area of real space. The system includes logic that maintains data representing in-stock items that match cells within multiple cells. The system determines the misplaced goods by comparing the data representing the inventory matched with the cell with the placement of the goods in stock at the stock position specified in the planogram. FIG. 14 shows a flow chart for determining the compliance of a planogram using a realogram. The process starts in step 1402. At step 1404, the system searches for a realogram for inventory item "i" at "t" when scoring. The score of the inventory item "i" in all cells in the realogram is compared to the placement of the inventory item "i" in the planogram (step 1406). If the realogram shows a SKU score for inventory item "i" that exceeds the threshold in cells that do not match the placement of inventory item "i" in the planogram (step 1408), the system identifies these items as misplacement. do. A warning or notification about an item that does not match the placement of the item in stock in the planogram is sent to the store employee who can take the misplaced item from its current position and put it back in the specified inventory position. (Step 1410). If the misplaced merchandise is not identified in step 1408, processing steps 1404 and 1406 are repeated for the next inventories "i".

In one embodiment, the store app displays the location of the product on the store map and guides the store employee to the misplaced product. Following this, the store app can display the correct position of the product on the store map, guide the employee to the correct part of the shelf, and place the product in the specified position. In another embodiment, the store app can also guide customers to inventory items based on the shopping list entered in the store app. The store app can use the real-time position of the inventory to guide the customer to the inventory location closest to the inventory on the map. In this example, the closest position of the in-stock item may be the position of the misplaced item that is not placed in the inventory position according to the store's planogram. FIG. 14B shows an exemplary user interface that displays a warning notification for misplaced inventory product “i” on the user interface display 1102. As mentioned above in FIG. 13B, various types of computing devices and alert notification mechanisms can be used to transmit this information to store employees.

[Improved inventory product forecast accuracy]

Another application of realograms is to improve the prediction of in-stock items by image recognition engines. The flowchart of FIG. 15 shows exemplary processing steps for adjusting inventory forecasts using realograms. The process starts at step 1502. At step 1504, the system receives the predicted confidence score probability of the product "i" from the image recognition engine. As mentioned above, WhatCNN is an exemplary image recognition engine that identifies in-stock items in the hands of a subject (or customer). WhatCNN outputs the predicted confidence score (or confidence value) probability of the inventories. At step 1506, the confidence score probability is compared to the threshold. If the probability value exceeds the threshold and indicates a higher confidence in the prediction (step 1508), the process is repeated for the next inventory item "i". Otherwise, if the confidence score probability is less than the threshold, processing continues with step 1510.

The realogram of the inventory item "i" at scoring "t" is searched for in step 1510. In one example, this may be the latest realogram, and in another example, the realogram at "t" that is temporally matched or closer to the time of the inventory event is retrieved from the realogram database 170. Can be done. At step 1512, the SKU score of the stock item "i" at the position of the stock event is compared to the threshold. When the SKU score exceeds the threshold value (step 1514), the prediction of the inventory product "i" by image recognition is accepted (step 1516). Update the customer log data structure associated with the inventory event accordingly. If the inventory event is a "take" event, the inventory item "i" is added to the customer's log data structure. If the inventory event is a "put" event, the inventory item "i" is removed from the customer's log data structure. If the SKU score is below the threshold (step 1514), the image recognition engine's prediction is rejected (step 1518). If the inventory event is a "take" event, as a result, the inventory item "i" is not added to the customer's log data structure. Similarly, if the inventory event is "put", as a result, the inventory item "i" is not removed from the customer's log data structure. The process ends in step 1520. In another embodiment, the SKU score of the inventory product "i" can be used to adjust the input parameters to the image recognition engine for determining the product prediction confidence score. WhatCNN, a convolutional neural network (CNN), is an example of an image recognition engine that predicts inventory products.

[Network Configuration]

FIG. 16 shows the architecture of a network hosting a store realogram engine 190 hosted on network node 106. The system includes a plurality of network nodes 101a, 101b, 101n, and 102 in the illustrated embodiment. In such an embodiment, the network node is also referred to as a processing platform. Processing platforms (network nodes) 103, 101a-101n, and 102, as well as cameras 1612, 1614, 1616, ... 1618 are connected to network 1681. A similar network hosts the store inventory engine 180 hosted on network node 104.

FIG. 13 shows a plurality of cameras 1612, 1614, 1616, ... 1618 connected to the network. Many cameras can be deployed in a particular system. In one embodiment, cameras 1612-1618 are connected to network 1681 using Ethernet®-based connectors 1622, 1624, 1626, and 1628, respectively. In such an embodiment, the Ethernet-based connector has a data transfer rate of 1 gigabit / sec, also referred to as Gigabit Ethernet. It should be appreciated that in other embodiments, the camera 114 is connected to the network using another type of network connection that can have faster or slower data transfer rates than Gigabit Ethernet®. .. Also, in an alternative embodiment, a set of cameras can be directly connected to each processing platform and the processing platform can be coupled to the network.

The storage subsystem 1630 stores basic programming and data structures that provide the functionality of a particular embodiment of the invention. For example, various modules that perform the functions of the store realogram engine 190 can be stored in the storage subsystem 1630. The storage subsystem 1630 is an example of a computer-readable memory comprising a non-temporary data storage medium, which is described herein including logic for calculating a real-space area realogram by the processing described herein. It comprises computer instructions stored in a computer-executable memory for performing all or any combination of data processing and image processing functions to be performed. In another example, computer instructions can be stored in other types of memory, including portable memory, including computer-readable non-temporary data storage media or media.

These software modules are typically run by the processor subsystem 1650. The host memory subsystem 1632 typically includes a main random access memory (RAM) 1634 for storing instructions and data during program execution, and a read-only memory (ROM) 1636 for storing fixed instructions. Includes some memory including. In one embodiment, RAM 1634 is used as a buffer for storing point cloud data structure tuples generated by the store realogram engine 190.

The file storage subsystem 1640 provides permanent storage for programs and data files. In one exemplary embodiment, the storage subsystem 1640 comprises four 120 gigabytes (GB) solid state disks (SSDs) in a RAID 0 (redundant array of independent disks) configuration identified by number 1642. In the exemplary embodiment, the data in the map database 140, the inventory event data in the inventory event database 150, the inventory data in the inventory database 160, and the realogram in the realogram database 170 not in RAM. The data is stored in RAID0. In the exemplary embodiment, the hard disk drive 1646 has a slower access speed than the RAID0 1642 storage. Solid State Disk (SSD) 1644 contains an operating system and related files for the store realogram engine 190.

In an exemplary configuration, four cameras 1612, 1614, 1616, 1618 are connected to a processing platform (network node) 103. Each camera has dedicated graphics processing units GPU1 1662, GPU2 1664, GPU3 1666, and GPU4 1668 to process the images sent by the cameras. It is understood that less than three or more cameras can be connected per processing platform. Thus, fewer or more GPUs are configured within the network node so that each camera has a dedicated GPU for processing image frames received from the camera. The processor subsystem 1650, the storage subsystem 1630, and the GPUs 1662, 1664, and 1666 communicate using the bus subsystem 1654.

The network interface subsystem 1670 is connected to the bus subsystem 1654 which forms part of the processing platform (network node) 104. The network interface subsystem 1670 provides an interface to an external network, including an interface to the corresponding interface device in other computer systems. The network interface subsystem 1670 allows the processing platform to communicate over the network, either by cable (or wiring) or wirelessly. Several peripheral devices, such as user interface output devices and user interface input devices, are also connected to the bus subsystem 1654, which forms part of the processing platform 104. These subsystems and devices are not intentionally shown in FIG. 13 to improve the clarity of the description. Although the bus subsystem 1654 is schematically shown as a single bus, alternative embodiments of the bus subsystem can use multiple buses.

In one embodiment, the camera 114 is a varifocal lens with a resolution of 1288 x 964, a frame rate of 30 FPS, and a working distance of 300 mm to infinity at 1.3 megapixels / image, 98.2 ° to 23. It can be mounted using a Cameraleon3 1.3 MP Color USB3 Vision (Sony ICX445) with a field of view with an 8 ° 1/3 inch sensor.

The techniques described herein also include a system for tracking inventory items in a real space area that includes an inventory display structure, along with corresponding methods and computer program products, above the inventory display structure. Multiple cameras or other sensors placed and sensors in multiple sensors that generate each image sequence of inventory display structures in the corresponding field of view in real space, provided that the field of view of each sensor is multiple. A memory that overlaps the field of view of at least one other sensor in the sensor and stores the dataset, where the dataset defines multiple cells with coordinates within an area of real space, to multiple sensors. Combined with a processing system, the processing system processes an image sequence generated by multiple sensors to locate an inventory event in three dimensions within an area of real space and responds to the inventory event. It contains logic to determine the closest cell in the dataset based on the location of the inventory event, and the processing system is associated with an inventory event that has a position to match a particular cell using each count of inventory events. Includes logic to calculate the score at the time of scoring for in-stock items. The system can include logic to select a set of inventories per cell based on the score. An inventory event can include a product identifier, an indicator to place or take, a position represented by a position along three axes of an area in real space, and a time stamp. The system can contain a dataset that defines multiple cells represented as a two-dimensional grid with coordinates within an area of real space, where the cells correlate with the front view portion of the stock location and the processing system Includes logic to determine the closest cell based on the location of the inventory event. The system can contain a dataset that defines multiple cells represented as a three-dimensional grid with coordinates within an area of real space, where the cells correlate with a portion of the volume on the stock location and the processing system Includes logic to determine the closest cell based on the position of the inventory event. The place indicator can identify that the item has been placed in the inventory position, and the place indicator identifies that the item has been removed from the inventory position. The logic that processes the image sequences generated by multiple sensors includes an image recognition engine that produces a dataset that represents the elements in the image that correspond to the hand, and analyzes the dataset from the image sequences from at least two sensors. Can be executed to determine the position of the inventory event in three dimensions. The image recognition engine can include a convolutional neural network. The logic for calculating the score of a cell uses the total of placing and taking of inventory items weighted by the separation between the putting and taking timestamps and the time of scoring, and stores the score in memory. be able to. The logic that determines the closest cell in the dataset based on the location of the inventory event is to calculate the distance from the location of the inventory event to the cell in the dataset, and to cell the inventory event based on the calculated distance. You can perform steps including matching with

Any data structure and code described above or referenced above can be any device or medium capable of storing the code and / or data used by the computer system according to many embodiments, non-temporarily. Stored in computer-readable memory, including a typical computer-readable storage medium. These include volatile memory, non-volatile memory, application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), disk drives, magnetic tapes, CDs (compact discs), DVDs (digital versatile). It includes, but is not limited to, magnetic and optical storage devices such as discs or digital video discs), or other media capable of storing computer-readable media currently known or developed in the future.

The preceding description is presented to enable the use and implementation of the disclosed technology. Various variations to the disclosed embodiments are obvious, and the principles defined herein can be applied to other embodiments and applications without departing from the spirit and scope of the disclosed art. Accordingly, the disclosed techniques are not intended to be limited to the embodiments shown, and should be given the broadest scope consistent with the principles and features disclosed herein. The scope of the disclosed technology is defined by the appended claims.

Claims (30)

A system for tracking inventory items in a real space area.
A plurality of sensors and a processing system coupled to the plurality of sensors are provided.
Sensors within the plurality of sensors generate an image sequence for each of the corresponding fields of view in the real space, where in the plurality of sensors the field of view of each sensor is with the field of view of at least one other sensor. Overlapping,
The processing system uses the image sequence generated by at least two of the plurality of sensors to identify an inventory event and, in response to the inventory event, inventory within the real space area. A system that includes logic to track the location of merchandise and count the inventories within said location. The system of claim 1, wherein the inventory event comprises a product identifier, an indicator to place or take, a position represented by a position in three dimensions of the real space area, and a time stamp. A data set that defines a plurality of cells having coordinates in the real space area is included in the memory.
The system according to claim 1, further comprising a logic that matches the position of an in-stock item with the coordinates of a cell and maintains data representing an in-stock item that matches the cells in the plurality of cells. The real space area contains multiple inventory locations and
The system according to claim 3, wherein the system includes a logic for matching an inventory position with the coordinates of cells in a plurality of cells that correlate with the inventory positions in the plurality of inventory positions. The system according to claim 3, wherein the processing system includes a logic for calculating a score based on each count of inventory events at the time of scoring for an inventory product having a position matching a specific cell. The logic for calculating the score of a cell uses the sum of placing and taking the inventory, weighted by the separation between the time stamp of placing and taking the inventory and the time of scoring. Item 5. The system according to item 5. The system according to claim 5, wherein the processing system includes a logic for rendering a cell in the plurality of cells and a display image representing the score of the cell. The system according to claim 5, wherein the processing system includes a logic for selecting a set of inventories for each cell based on the score. The real space area contains a plurality of inventory positions, and the coordinates of the cells in the plurality of cells correlate with the inventory positions in the plurality of inventory positions.
A data set that defines a plurality of cells having coordinates in the real space area is contained in the memory, and a planogram that specifies the arrangement of inventory products at the stock position in the real space area is contained in the memory.
The logic that maintains the data representing the inventory items that match the cells in the plurality of cells compares the data representing the inventory that matches the cells with the placement of the inventory items within the inventory position specified in the planogram. The system according to claim 3, comprising logic for determining misplaced goods by doing so. The real space area contains a plurality of inventory positions, and the coordinates of the cells in the plurality of cells correlate with the inventory positions in the plurality of inventory positions.
For a specific inventory product matched with a specific cell, the logic for maintaining the data representing the inventory product matching the cells in the plurality of cells counts the specific inventory product in the cell. The system of claim 3, comprising logic for determining whether the inventory item on an inventory position that correlates with a particular cell is below the threshold for restocking. A method for tracking inventory items in a real space area.
Using a plurality of sensors to generate an image sequence for each of the corresponding fields of view in the real space, where the field of view of each sensor overlaps the field of view of at least one other sensor.
The image sequence generated by at least two of the plurality of sensors is used to identify inventory events and.
A method of tracking the location of inventory goods within the real space area and counting the inventory goods within the location in response to the identification of the inventory event. 11. The method of claim 11, wherein the inventory event comprises a product identifier, an indicator to place or take, a position represented by a position along three axes of the real space area, and a time stamp. Specifying multiple cells with coordinates within the real space area,
To store the plurality of cells in a data set in memory,
Matching the position of in-stock items with the cell coordinates, and
11. The method of claim 11, further comprising maintaining data representing inventories that match cells in the plurality of cells. The real space area contains multiple inventory locations and
13. The method of claim 13, wherein the coordinates of the cells in the plurality of cells correlate with the inventory positions in the plurality of inventory positions. 13. The method of claim 13, further comprising calculating a score using each count of inventory events at the time of scoring for an inventory item having a position that matches a particular cell. Using the sum of placing and taking the inventory, weighted by the separation between the time stamp of placing and taking the inventory and the time of scoring, to calculate the score of the cell, and The method of claim 15, further comprising storing the score in memory. 15. The method of claim 15, further comprising rendering a cell within the plurality of cells and a display image representing the score of the cell. 15. The method of claim 15, further comprising selecting a set of inventories for each cell based on the score. The real space area contains a plurality of inventory positions, and the coordinates of the cells in the plurality of cells correlate with the inventory positions in the plurality of inventory positions.
Specifying a plurality of cells having coordinates in the area of the real space, and storing the specified cells as a data set in the memory.
Specifying the arrangement of inventory products at the inventory position in the real space area as a planogram and storing the planogram in the memory.
Maintaining data representing in-stock items that match cells in the plurality of cells, and
13. The method of claim 13, further comprising determining misplaced merchandise by comparing the data representing the inventory matching the cell with the placement of the inventories in the inventory position. The real space area contains a plurality of inventory positions, and the coordinates of the cells in the plurality of cells correlate with the inventory positions in the plurality of inventory positions.
Maintaining the data representing inventories that match cells in the plurality of cells, and
For an inventory item that matches a particular cell, whether the count of the particular inventory item in that cell is lower than the threshold for restocking the inventory item on the inventory position that correlates with the particular cell. 13. The method of claim 13, further comprising determining. A non-temporary computer-readable storage medium that stores computer program instructions for tracking inventory items in a real-space area.
The method implemented when the instruction is executed on the processor is
Using a plurality of sensors to generate an image sequence for each of the corresponding fields of view in the real space, where the field of view of each sensor overlaps the field of view of at least one other sensor.
The image sequence generated by at least two of the plurality of sensors is used to identify inventory events and.
A non-temporary computer-readable storage medium comprising tracking the location of inventory goods within the real space area and counting the inventory goods within said location in response to said identification of the inventory event. 21. The non-temporary computer-readable storage medium of claim 21, wherein the inventory event comprises a product identifier, an indicator to place or take, a position represented by a position along three axes of the real space area, and a time stamp. .. The implementation of the above method
Specifying multiple cells with coordinates within the real space area,
To store the plurality of cells in a data set in memory,
Matching the position of in-stock items with the cell coordinates, and
21. The non-temporary computer-readable storage medium of claim 21, further comprising maintaining data representing in-stock items that match the cells in the plurality of cells. The real space area contains multiple inventory locations and
23. The non-temporary computer-readable storage medium according to claim 23, wherein the coordinates of the cells in the plurality of cells correlate with the inventory positions in the plurality of inventory positions. The implementation of the above method
24. The non-temporary computer-readable storage medium of claim 24, further comprising calculating a score using each count of inventory events at the time of scoring for an inventory item having a position matching a particular cell. The implementation of the above method
Using the sum of placing and taking the inventory, weighted by the separation between the time stamp of placing and taking the inventory and the time of scoring, to calculate the score of the cell, and 25. The non-temporary computer-readable storage medium according to claim 25, further comprising storing the score in memory. The implementation of the above method
25. The non-temporary computer-readable storage medium of claim 25, further comprising rendering a cell within the plurality of cells and a display image representing the score of the cell. The implementation of the above method
25. The non-temporary computer-readable storage medium of claim 25, further comprising selecting a set of in-stock items for each cell based on the score. The real space area contains a plurality of inventory positions, and the coordinates of the cells in the plurality of cells correlate with the inventory positions in the plurality of inventory positions.
The implementation of the above method
Specifying a plurality of cells having coordinates in the area of the real space, and storing the specified cells as a data set in the memory.
Specifying the arrangement of inventory products at the inventory position in the real space area as a planogram and storing the planogram in the memory.
Maintaining data representing in-stock items that match cells in the plurality of cells, and
23. The non-temporary computer-readable storage according to claim 23, further comprising determining a misplaced product by comparing the data representing the inventory matching the cell with the placement of the stock product in the stock position. Medium. The real space area contains a plurality of inventory positions, and the coordinates of the cells in the plurality of cells correlate with the inventory positions in the plurality of inventory positions.
The implementation of the above method
Maintaining the data representing inventories that match cells in the plurality of cells, and
For an inventory item that matches a particular cell, whether the count of the particular inventory item in that cell is lower than the threshold for restocking the inventory item on the inventory position that correlates with the particular cell. 23. The non-temporary computer-readable storage medium according to claim 23.

Images