TWI773797B - System, method and computer program product for tracking multi-joint subjects in an area of real space - Google Patents

Abstract

Systems and techniques are provided for tracking puts and takes of inventory items by subjects in an area of real space. A plurality of cameras with overlapping fields of view produce respective sequences of images of corresponding fields of view in the real space. In one embodiment, the system includes first image processors, including subject image recognition engines, receiving corresponding sequences of images from the plurality of cameras. The first image processors process images to identify subjects represented in the images in the corresponding sequences of images. The system includes second image processors, including background image recognition engines, receiving corresponding sequences of images from the plurality of cameras. The second image processors mask the identified subjects to generate masked images. Following this, the second image processors process the masked images to identify and classify background changes represented in the images in the corresponding sequences of images.

Description

System, method and computer program product for tracking a multi-joint body in an area of real space

The present invention relates to a system and components thereof that can be used for cashierless checkout.

The difficult problem of image processing occurs when images from many cameras, which are deployed over a large space, are used to recognize and track the movements of the subject.

Tracking the movements of subjects within areas of real space, such as people in shopping stores, poses a number of technical challenges. For example, consider such an image processing system deployed in a shopping store, where most consumers move in aisles between shelves and in open spaces within a shopping store. Consumers remove items from the shelves and place those items into their respective shopping carts or baskets. Consumers can also put an item on the shelf if they don't want the item.

While the consumer is performing these actions, different parts of the consumer and different parts of the shelves or other display structures holding the store's inventory will be blocked from images from different cameras due to other consumers, shelves, and product displays. The existence of waiting. At the same time, there may be many consumers in the store at any given moment, making it difficult to identify and track individuals and their movements over time.

It would be desirable to provide a system that more efficiently and automatically recognizes and tracks the putting and taking actions of subjects in large spaces; and performs other procedures that support complex interactions of subjects with their environment, including such as cashierless checkout, etc. Function.

A system, and method for operating the system, is provided to track changes by subjects (such as people) in areas of real space, and other complex interactions of subjects with their environment, using image processing. This function of tracking changes by image processing creates complex problems in computer engineering regarding the type of image data to be processed, what processing of image data should be performed, and how to determine actions from image data with high reliability. The system described herein can only use images from cameras above it deployed in real space, so that it does not require retrofitting of store shelves and floor space with sensors, etc. for deployment in a given setting.

Provided is a system and method for tracking the placement and removal of inventory items by subjects in an area of physical space that includes inventory display structures, comprising using a plurality of cameras disposed on the inventory display structures to generate the inventory display structures The respective sequences of images of these inventory display structures are in this real In a corresponding viewing field in space, the viewing field of each camera overlaps the viewing field of at least one other camera of the plurality of cameras. Using these sequences of images, a system and method is described for detecting the dropping and removal of inventory items by semantically identifying significant changes in the sequences of images of the inventory items that are associated with them on an inventory display structure; and The semantically significant changes are associated with the subjects represented in the sequences of images.

Provided is a system and method for tracking the placement and removal of inventory items by subjects in an area of real space, comprising using a plurality of cameras disposed on the inventory display structures to generate an image of the inventory display structures The respective sequences of images are in corresponding viewing domains in the real space, the viewing domains of each camera overlapping the viewing domains of at least one other camera of the plurality of cameras. Using these sequences of images, a system and method is described for detecting the drop and removal of inventory items by processing foreground data in the sequences of images to identify gestures of subjects and inventory items associated with the gestures.

Also, a system and method are described that combine foreground processing and background processing in the same sequence of images. In this combination, the systems and methods provided include using the sequences of images to detect inventory by processing foreground data in the sequences of images to identify gestures of the subject and inventory items associated with the gestures the dropping and taking of items; and the use of these sequences of images to detect the dropping and removal of inventory items by semantically identifying significant changes in those sequences of images of the inventory items in their relation to the inventory display structure, and The semantically significant changes are associated with the subjects represented in the sequences of images.

In the embodiments described herein, the system uses a plurality of cameras to generate individual sequences of images of corresponding viewing domains in the real space. The viewing field of each camera overlaps the viewing field of at least one other camera of the plurality of cameras. The system includes a first image processor including a subject image recognition engine that receives a corresponding sequence of images from the plurality of cameras. The first image processors process images to identify subjects represented in the images in the corresponding sequences of images. The system further includes a second image processor including a background image recognition engine that receives a corresponding sequence of images from the plurality of cameras. The second image processors mask the identified subjects to generate masked images, and process the masked images to identify and classify background changes represented in the images in the corresponding sequences of images.

In one embodiment, the background image recognition engines include convolutional neural networks. The system includes logic to cause the identified context change to be associated with the identified subject.

In one embodiment, the second image processors include background image storage for storing background images of corresponding sequences of images. The second image processors further include masking logic for processing images in the sequences of images to replace foreground image data and background image data representing the identified subjects. The background image data is collected from the background images of the corresponding sequences of images to provide the masked images.

In one embodiment, the masking logic combines sets of N masked images in the sequences of images to generate a sequence of factorized images for each camera. The second image processors process the factorized image by of this sequence to identify and classify background changes.

In one embodiment, the second image processors include logic to generate altered data structures for the corresponding sequences of images. The change data structures include the coordinates in the masked image of the identified background changes, the identifiers of the inventory item bodies of the identified background changes, and the category of the identified background changes. The second image processors further include coordination logic to process changing data structures from sets of cameras with overlapping viewing fields to find the identified background changes in real space.

In one embodiment, the categories of identified background changes in the change data structures indicate whether the identified inventory item was added or removed relative to the background image.

In another embodiment, the categories of identified background changes in the change data structures indicate whether the identified inventory item was added or removed relative to the background image. The system further includes logic to associate a context change with the identified subject. Finally, the system includes logic to perform detection of the removal of inventory items by the identified entities and detection of the placement of inventory items by the identified entities on the inventory display structure.

In another embodiment, the system includes logic to cause a context change to be associated with an identified subject. The system further includes logic to perform detection of the removal of inventory items by the identified entities and detection of the placement of inventory items by the identified entities on the inventory display structure.

The system may include a third image processor as described herein, including a foreground image recognition engine, receiving a corresponding sequence of images from the plurality of cameras. The third image processors process images to identify and Foreground changes represented in the images in the corresponding sequences of classified images.

A system and method for operating an operating system are provided for tracking a multi-joint subject (such as a human) in real space. The system uses a plurality of cameras to generate individual sequences of images of corresponding viewing domains in the real space. The viewing field of each camera overlaps the viewing field of at least one other camera of the plurality of cameras. The sequence processes images in the sequences of images to generate an array of joint data structures corresponding to each image. The array of joint data structures corresponding to a particular image classifies the elements of a particular image by the joint type, the time of the particular image, and the coordinates of the elements in the particular image. The system then transforms the coordinates of the elements in the array of joint data structures corresponding to the images in the different sequences into candidate joints with coordinates in real space. Finally, the system identifies clusters of candidate joints, wherein the clusters comprise respective sets of candidate joints with coordinates in real space, becoming a multi-joint body in real space.

In one embodiment, the image recognition engines include convolutional neural networks. The processing of the image by the image recognition engine includes a confidence array of elements that generate the image. The confidence array for a particular element of the image includes confidence values for a plurality of joint types for that particular element. The confidence arrays are used to select the joint type of the joint data structure of the particular element according to the confidence array.

In one embodiment of the system for tracking a multi-joint subject, identifying the set of candidate joints includes applying a heuristic function to identify the candidate joints based on physical relationships between the subject's joints in real space. The assembly is a multi-joint body. The process includes storing the sets of joints that are identified as multi-joint bodies. Identifying the set of candidate joints includes determining whether a candidate joint identified in an image taken at a particular time is a member of one of the sets of candidate joints that were identified as candidate joints for a multi-joint subject in a previous image.

In one embodiment, the sequences of images are synchronized such that the images in each of the sequences of images captured by the plurality of cameras represent the time scale of the subject's movement through the space. Real space at a single point in time.

A coordinate system in real space that is identified as a member of the set of candidate joints of a multi-joint body identifies a location in the region of the multi-joint body. In some embodiments, the processing includes simultaneous tracking of the positions of the plurality of articulated bodies in the region of real space. In some embodiments, the processing includes determining when one of the plurality of articulated bodies leaves the region of real space. In some embodiments, the processing includes determining the direction in which the articulated body is facing at a given point in time. In the embodiments described herein, the system uses a plurality of cameras to generate individual sequences of images of corresponding viewing domains in the real space. The viewing field of each camera overlaps the viewing field of at least one other camera of the plurality of cameras. The system processes images in the sequences of images it receives from the plurality of cameras to identify subjects represented in the images and generate categories of the identified subjects. Finally, the system processes the categories of identified subjects of the set of images in the sequences of images to detect the taking of inventory items by the identified subjects and the placing of inventory items on shelves by the identified subjects.

In one embodiment, the category identifies whether the identified entity holds inventory items. The class also identifies whether the identified subject's hand is close to the shelf or whether the identified subject's hand is close to the identified subject. The category of whether the hand is close to the identified subject may include whether the identified subject's hand is close to a basket associated with the identified subject, and whether it is close to the body of the identified subject.

With the described techniques, images representing the hands of the subject in the viewing field can be processed to generate categories of the subject's hands in the plurality of images in the time series. The class of hands from the sequence of images can be processed, using a convolutional neural network (in some embodiments), to identify movements by the subject. These actions can be the dropping and removal of inventory items (as described in the embodiments described herein), or other types of actions that can be decrypted by processing the image of the hand.

With the described techniques, images are processed to identify subjects in the viewing field and to locate the joints of those subjects. The location of the subject's joints can be processed as described herein to identify bounding boxes in corresponding images that include the subjects' hands. The data within the bounding boxes may be the processed class of the subject's hand in the corresponding image. The class of hands from an identified subject (which was generated from the sequence of images in this way) can be processed to identify movements by the subject.

In a system that includes a plurality of image recognition engines, such as both foreground and background image recognition engines, the system may perform: detection of inventory items removed by the identified subjects and placement of inventory items in inventory by the identified subjects A first set of detections on the structure is shown, and by these A second set of detections of removal of inventory items by identified entities and detections of inventory items being put down by those identified entities on an inventory display structure. Selection logic to process the first and second sets of detections may be used to generate a log data structure. The log data structure includes a list of inventory items for identified entities.

In the embodiments described herein, the sequences of images from cameras of the plurality of cameras are synchronized. The same camera and the same sequence of images are used by both the foreground and background image processors in a preferred embodiment. As a result, redundant detection of put-down and take-out of inventory items is performed using the same input data, allowing high confidence (and high accuracy) in the resulting data.

In one technique described herein, the system includes logic to detect the placement and removal of inventory items by identifying gestures of the subject and inventory items associated with the gestures represented in the sequences of images. This can be done using a foreground image recognition engine in conjunction with a subject image recognition engine as described herein.

In another technique described herein, the system includes logic to detect the dropping and removal of inventory items by semantically identifying the placement and removal of inventory items on an inventory display structure, such as a shelf, over time. Significant changes and associated semantically significant changes with subjects represented in the sequences of images. This can be accomplished using a background image recognition engine in conjunction with a subject image recognition engine as described herein.

When applying the system described herein, both gesture analysis and semantic difference analysis can be combined and performed on the same data from an array of cameras. step on the same sequence of images.

Methods and computer program products, which can be executed by computer systems, are also described herein.

Other aspects and advantages of the present invention can be found in a review of the drawings, detailed description, and the scope of claims, which continue below.

100: System

101a, 101b, 101c: network nodes

102: Network Node

110: Tracking Engine

112a-112n: Image recognition engine

114: Camera

116a: Walkway

116b: Walkway

116n: Walkway

120: Adjuster

140: Subject Database

150: Training database

160: Heuristics Database

170: Tuning Database

181: Internet

202: Shelf A

204: Shelf B

206: Camera A

208: Camera B

216: Viewing Domain

218: Viewing Domain

220: Flooring

230: Roof

412-418: Cameras

422-428: Ethernet-Based Connectors

430: Storage Subsystem

432: Host Memory Subsystem

434: Random Access Memory (RAM)

436: Read Only Memory (ROM)

440: File Storage Subsystem

442: RAID 0

444: Solid State Drive (SSD)

446: Hard Disk Drive (HDD)

450: Processor Subsystem

454: Busbar Subsystem

462: GPU 1

464: GPU 2

466: GPU 3

481: Internet

510: Input image

520: Filter

530: Convolutional layer

540: output matrix

702: Overall Metric Calculator

800: Main data structure

1411: Video Program

1415: Scene Program

1452: output

1453: input

1457: output

1502: Circular buffer

1504: Bounding Box Generator

1506: WhatCNN

1508:WhenCNN

1510: Shopping Cart Data Structure

1520: Video channel

1522: Coordination Logic Module

2002: Heuristics

2111: input

2113: The first convolutional layer

2115: Second convolutional layer

2117: Square Box

2119, 2121: Layer

2123: Eight convolutional layers

2125, 2127: Convolutional layer

2129: Eight convolutional layers

2135: Fully connected layer

2210: "One-handed" model

2212: Convolutional layer

2214: Aggregate layer

2216: Block 0

2218: Block 1

2220: Block 2

2222: Block 3

2310: Square Box

2312: Convolutional layer

2314: Batch normalization layer

2316: ReLU nonlinearity

2318:conv1

2320:conv2

2322:conv3

2324: addition operation

2326: skip connection

2410: Fully Connected Layer (FC)

2412: Reshaping Operators

2420: next level

2422:MatMul

2424: Operator

2426: output

2502: Part 1

2504: Part II

2506: Part Three

2508: Part Four

2510: Part V

2512: Part VI

2602: First Image Processor Subsystem

2604: Second Image Processor Subsystem

2606: Third Image Processor Subsystem

2608: Select Logic Components

2702: Mask Logic Components

2704: Background image storage

2706: Factorized Imagery

2710: Bit Mask Calculator

2714a-2714n: Changes to CNN

2718: Coordination Logic Components

2720: log generator

2724: Mask Generator

Figure 1 illustrates an architectural high-level overview of a system in which a tracking engine uses joint data generated by an image recognition engine to track a subject.

Figure 2 is a side view of an aisle in a shopping store illustrating the camera configuration.

3 is a top view of the aisle of FIG. 2 in a shopping store illustrating the camera configuration.

FIG. 4 is a camera and computer hardware configuration configured to host the image recognition engine of FIG. 1 .

FIG. 5 illustrates a convolutional neural network illustrating the recognition of joints in the image recognition engine of FIG. 1 .

Figure 6 shows an example data structure for storing joint information.

Figure 7 illustrates the tracking engine of Figure 1 with an overall metric calculator.

FIG. 8 shows an example data structure for a body used to store information including related joints.

FIG. 9 is a flowchart illustrating the procedural steps of tracking a subject by the system of FIG. 1 .

FIG. 10 is a flowchart showing more detailed procedural steps of the camera calibration step of FIG. 9 .

FIG. 11 is a flowchart showing more detailed procedural steps of the video procedural steps of FIG. 9 .

FIG. 12A is a flowchart showing the first part of the more detailed program steps of the scenario program of FIG. 9 .

FIG. 12B is a flow chart showing the second part of the more detailed program steps of the scenario program of FIG. 9 .

13 is an illustration of an environment in which an embodiment of the system of FIG. 1 is used.

FIG. 14 is an illustration of a video and scene program in an embodiment of the system of FIG. 1 .

Figure 15A is a schematic diagram showing a pipeline with multiple convolutional neural networks (CNNs) including articulated CNN, WhatCNN, and WhenCNN to generate a per-agent shopping cart data structure in real space.

Figure 15B shows most image channels from most cameras and the coordination logic for the subjects and their respective shopping cart data structures.

Figure 16 is a flow chart illustrating the procedural steps for identifying and updating subjects in real space.

Figure 17 is a flow chart showing procedural steps for processing the subject's hand joints to identify inventory items.

Figure 18 is a flow chart showing inventory for each hand joint Procedural steps for time series analysis of items to generate a shopping cart data structure for each subject.

Figure 19 is an illustration of the WhatCNN model in an embodiment of the system of Figure 15A.

20 is an illustration of the WhenCNN model in an embodiment of the system of FIG. 15A.

Figure 21 presents an example architecture of a WhatCNN model that identifies the dimensions of the convolutional layers.

Figure 22 presents a high-level block diagram of an embodiment of WhatCNN for the class of hand images.

FIG. 23 presents details of the first block of the high-order block diagram of the WhatCNN model proposed in FIG. 22 .

Figure 24 presents the operators in the fully connected layer in the example WhatCNN model presented in Figure 22.

Figure 25 is an example name of an image file stored as part of the training data set for the WhatCNN model.

Figure 26 is a high-level architecture of a system for tracking changes by subject in an area of real space, where the selection logic is between a first detection using background semantic diffing and a redundant detection using foreground area proposals choice between.

FIG. 27 presents the components of the subsystems which implement the system of FIG. 26. FIG.

Figure 28A is a flowchart showing the first part of the detailed procedural steps for determining inventory events and generation of a shopping cart data structure.

Figure 28B is a flowchart showing the second part of the detailed procedural steps for determining the inventory event and the generation of the shopping cart data structure.

The following description is presented to enable any person skilled in the art to make and use the present invention, and is provided in the context of a particular application and its requirements. Various modifications to the disclosed embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown but is to be accorded the widest scope consistent with the principles and features disclosed herein.

System Overview

One system and various embodiments of the main technology are described with reference to Figures 1-28A/28B. The system and procedures are described with reference to FIG. 1 , an architectural level overview of the system according to an embodiment. Since FIG. 1 is an architectural diagram, some details are omitted to improve the clarity of the description.

The discussion of Figure 1 is as follows. First, the elements of the system are described, followed by their interconnection. Next, the use of the elements in the system is described in more detail.

FIG. 1 provides a block diagram level illustration of a system 100 . The system 100 includes a camera 114, a network node hosting the image recognition engines 112a, 112b, and 112n, a tracking engine 110 deployed in the network node (or nodes) on the network, a calibrator 120, a subject database 140, training database 150. Heuristics database 160 (for joint heuristics, for drop and take heuristics, and other heuristics for coordinating and combining the output of most image recognition engines as described below), calibration database 170, and a communication network or network 181. A network node may host only one image recognition engine, or several image recognition engines, as described herein. The system may also include an inventory database and other supporting data.

As used herein, a network node is an addressable hardware device or virtual device attached to a network and capable of transmitting, receiving, or communicating information to or from other network nodes through a communication channel. Examples of electronic devices that can be deployed as hardware network nodes include all kinds of computers, workstations, laptops, handheld computers, and smart phones. Network nodes can be implemented in cloud-based server systems. More than one virtual device configured as a network node can be implemented using a single physical device.

For the sake of simplicity, only three network nodes that host the image recognition engine are shown in the system 100 . However, any number of network nodes hosting the image recognition engine may be connected to the tracking engine 110 through the network 181 . At the same time, the image recognition engine, tracking engine, and other processing engines described herein may be executed using more than one network node in a distributed architecture.

The interconnection of the elements of system 100 will now be described. Network 181 couples network nodes 101a, 101b, and 101c, respectively, which host image recognition engines 112a, 112b, and 112n, network node 102 that hosts tracking engine 110, tuner 120, and host data Library 140 , training database 150 , joint heuristics database 160 , and tuning database 170 . Camera 114 Connected to the tracking engine 110 through the network node hosting the image recognition engines 112a, 112b, and 112n. In one embodiment, cameras 114 are installed in a shopping store (such as a supermarket) such that a set of cameras 114 (two or more) with overlapping viewing fields are placed over each aisle to capture the reality in the store image of space. In FIG. 1, two cameras are disposed above the aisle 116a, two cameras are disposed above the aisle 116b, and three cameras are disposed above the aisle 116n. Camera 114 is mounted above the walkway with overlapping viewing fields. In this embodiment, the cameras are configured to target that a consumer moving in the aisles of a shopping store is present in the viewing fields of two or more cameras at any point in time.

The cameras 114 may be synchronized with each other in time so that their images are captured simultaneously (or close in time) and at the same image capture rate. The camera 114 may transmit respective continuous streams of images at a predetermined rate to the network nodes hosting the image recognition engines 112a-112n. The images captured in all cameras that cover an area of real space simultaneously (or close in time) are synchronized, since their synchronized images can be identified in the processing engine as representing a fixed position in real space different perspectives of the subject. For example, in one embodiment, the cameras transmit image frames at a rate of 30 frames per second (fps) to the respective network nodes hosting the image recognition engines 112a-112n. Each frame has a timestamp, an identification of the camera (abbreviated as "camera_id"), and a frame identification (abbreviated as "frame_id"), along with image data.

Cameras mounted above the walkway are connected to respective image recognition engines. For example, in FIG. 1, two cameras mounted above the walkway 116a are connected to the network node 101a that hosts the image recognition engine 112a. same Ground, the two cameras installed above the walkway 116b are connected to the network node 101b that hosts the image recognition engine 112b. Each image recognition engine 112a-112n hosted in a network node or nodes 101a-101n processes image frames received from one of the cameras in each of the illustrated examples, respectively.

In one embodiment, each image recognition engine 112a, 112b, and 112n is implemented as a deep learning algorithm, such as a convolutional neural network (abbreviated as CNN). In this embodiment, the CNN is trained using the training database 150 . In the embodiments described herein, image recognition of subjects in real space is based on identifying and clustering identifiable joints in the images, wherein groups of joints can be assigned to a respective subject. For this joint-based analysis, the training database 150 has a large collection of images for each different type of joint of the subject. In an exemplary embodiment of a shopping store, the subjects are consumers moving in aisles between shelves. In an exemplary embodiment, during the training of a CNN, the system 100 is referred to as a "training system." After using the training database 150 to train the CNN, the CNN is switched to a generation mode to process images of consumers in a shopping store in real time. In an exemplary embodiment, during generation, the system 100 is referred to as a runtime system (also known as an inference system). The CNN in each image recognition engine generates an array of the image's joint data structures in its respective stream of the image. In one embodiment as described herein, an array of joint data structures is generated for each processed image such that each of its image recognition engines 112a-112n generates an output stream of the array of joint data structures. These arrays of joint data structures from cameras with overlapping viewing fields are further processed to form groups of joints and to identify these groups of joints as hosts.

The camera 114 was calibrated before switching the CNN to production mode. The calibrator 120 calibrates the cameras and stores the calibration data in the calibration database 170 .

Tracking engine 110 (hosted on network node 102) receives a continuous stream of arrays of joint data structures from the bodies of image recognition engines 112a-112n. The tracking engine 110 processes the array of joint data structures and transforms the coordinates of elements in the array of joint data structures corresponding to images in different sequences into candidate joints with coordinates in real space. For each set of synchronized images, combinations of candidate joints identified throughout the real space may be considered (for analogy purposes) as galaxies like candidate joints. For each subsequent time point, the movement of the candidate joint is recorded so that its galaxy changes over time. The output of the tracking engine 110 is stored in the subject database 140 .

The tracking engine 110 uses logic to identify groups or sets of candidate joints with coordinates in real space as subjects in the real space. For analogy purposes, each set of candidate points is like a cluster of candidate joints at each point in time. The cluster of candidate joints can move over time.

The logic to identify the set of candidate joints includes heuristic functions based on physical relationships between subjects in real space. These heuristic functions are used to identify the set of candidate joints as subjects. Heuristic functions are stored in the heuristic database 160 . The output of the tracking engine 110 is stored in the subject database 140 . Thus, the sets of candidate joints include the respective candidate joints that have a relationship with the other respective candidate joints based on the heuristic parameters, and the given set of which has been identified (or can be identified) as the respective subject A subset of candidate joints in .

The actual communication path through network 181 may be point-to-point through public and/or private networks. Communication can occur over a variety of networks 182, such as private networks, VPNs, MPLS circuits, or the Internet; and can use appropriate application programming interfaces (APIs) and data exchange formats, such as representation state transfer (REST) ), JavaScript ^™ Object Notation (JSON), Extensible Markup Language (XML), Simple Object Access Protocol (SOAP), Java ^™ Message Service (JMS), and/or the Java Platform Module System. All of these communications can be encrypted. Communication is usually through networks such as LAN (Local Area Network), WAN (Wide Area Network), telephone network (Public Switched Telephone Network (PSTN)), Session Initiation Protocol (SIP), wireless network, peer-to-peer network, Star networks, token ring networks, hub networks, the Internet, including mobile Internet via protocols such as EDGE, 3G, 4G LTE, Wi-Fi, and WiMAX. Additionally, various authorization and authentication techniques such as username/passcode, Open Authorization (OAuth), Kerberos, SecureID, digital credentials, and more can be used to secure communications.

The techniques disclosed herein may be implemented in the context of any computer-implemented system, including database systems, multi-tenant environments, or relational database implementations, such as Oracle ^™ compliant relational implementations, IBM DB2 Enterprise Server ^™ A compatible relational implementation, a MySQL ^™ or PostgreSQL ^™ compatible relational implementation, or a Microsoft SQL Server ^™ compatible relational implementation; or a NoSQL ^™ non-relational database implementation such as a Vampire ^™ compatible Non-Associative Implementation, Apache Cassandra ^™ Compliant Non-Associative Implementation, BigTable ^™ Compliant Non-Associative Implementation, or HBase ^™ or DynamoDB ^™ Compliant Non-Associative Implementation. Furthermore, the disclosed techniques can be implemented using: different programming models, such as MapReduce ^™ , chunk synchronous programming, MPI primitives, etc.; or different scalable batch and stream management systems, such as Apache Storm ^™ , Apache Spark ^™ , Apache Kafka ^™ , Apache Flink ^™ , Truviso ^™ , Amazon Elasticsearch Service ^™ , Amazon Web Services ^™ (AWS), IBM Info-Sphere ^™ , Borealis ^™ , and Yahoo! ^S4TM .

Camera configuration

The camera 114 is configured to track a multi-joint monolith (or body) in three-dimensional (abbreviated 3D) real space. In an example embodiment of a shopping store, the real space may include an area of the shopping store in which sale items are stacked in shelves. A point in real space can be represented by an (x, y, z) coordinate system. Each point in the area of real space where the system is deployed is covered by the viewing field of two or more cameras 114 .

In a shopping store, shelving and other inventory display structures may be configured in a variety of ways, such as along the walls of the shopping store, or in rows forming aisles, or a combination of the two configurations. Figure 2 shows the arrangement of the shelves (which form the aisle 116a), seen from one end of the aisle 116a. Two cameras (camera A 206 and camera B 208) are placed above the aisle 116a at a predetermined distance from the roof 230 and floor 220 of the shopping store above an inventory display structure such as a shelf. Camera 114 includes a camera disposed above and having a viewing field that covers various parts of the inventory display structure and floor area in real space. camera. The coordinate system in real space of the members of the set of candidate joints identified as the subject identifies the position in the floor area of the subject. In the example embodiment of a shopping store, the real space may include all of the floors 220 in the shopping store from where inventory can be accessed. Camera 114 is positioned and oriented so that its area of floor 220 and shelves can be seen by at least two cameras. Camera 114 also covers at least a portion of shelves 202 and 204 and the floor space in front of shelves 202 and 204 . Camera angles are chosen to have both a steep view (straight down), and an angled view (which provides a more complete body image of the consumer). In an exemplary embodiment, cameras 114 are configured to be eight (8) feet tall or higher throughout the shopping store. Figure 13 presents an illustration of such an embodiment.

In FIG. 2, cameras 206 and 208 have overlapping viewing fields, which cover the space between shelf A 202 and shelf B 204, with overlapping viewing fields 216 and 218, respectively. A position in real space is represented as a (x, y, z) point in a real space coordinate system. "x" and "y" represent locations on a two-dimensional (2D) plane, which may be the floor 220 of a shopping store. The value "z" is the height of the point above the 2D plane on the floor 220 in one configuration.

3 illustrates walkway 116a as viewed from the top of FIG. 2, further showing an example configuration of the locations of cameras 206 and 208 above walkway 116a. Cameras 206 and 208 are placed closer to opposite ends of aisle 116a. Camera A 206 is positioned a predetermined distance from shelf A 202 and camera B 208 is positioned a predetermined distance from shelf B 204 . In another embodiment, where more than two cameras are placed above the walkway, the cameras are placed at equal distances from each other. In this embodiment, the two cameras are placed close to at the opposite end and a third camera is placed in the middle of the aisle. It should be understood that several different camera configurations are possible.

camera adjustment

Camera adjuster 120 performs two types of adjustments: internal and external. In the internal adjustment, the internal parameters of the camera 114 are adjusted. Examples of internal camera parameters include focus length, principal point, skew, fisheye coefficient, and the like. Various techniques for in-house camera calibration can be used. One such technique was proposed by Zhang in "A flexible new technique for camera calibration", which was published in IEEE Transactions on Pattern Analysis and Machine Intelligence, Volume 22, No. 11, November 2000.

In external tuning, external camera parameters are tuned to generate mapping parameters for transforming 2D image data into 3D coordinates in real space. In one embodiment, a subject, such as a person, is introduced into the real space. The subject moves through the real space, on a path through the viewing field of each camera 114 . At any given point in the real space, the host system appears in the viewing field of at least two cameras which form the 3D scene. However, the two cameras have different views of the same 3D scene in their respective two-dimensional (2D) image planes. Features in the 3D scene, such as the subject's left wrist, are viewed by two cameras at different locations in their respective 2D image planes.

Point correspondences are established between each pair of cameras and the overlapping viewing fields of a given scene. Because each camera has a different view of the same 3D scene, So a point corresponds to a two-pixel location (one location from each camera with overlapping viewing fields) that represents the projection of the same point in the 3D scene. A number of point correspondences are identified for each 3D scene using the results of the image recognition engines 112a-112n for external calibration purposes. The image recognition engine recognizes the positions of the joints as (x,y) coordinates, such as column and row numbers, of pixels in the 2D image plane of the respective camera 114 . In one embodiment, the joint is one of 19 different types of joints in the body. As the subject moves through the viewing fields of the different cameras, the tracking engine 110 receives its (x,y) coordinates for each of the subject's 19 different types of joints for calibration from the camera 114 of each image.

For example, consider the image from camera A and the image from camera B, both acquired at the same point in time and having overlapping viewing fields. There are pixels in the image from camera A that correspond to pixels in the synchronized image from camera B. Consider that there is a particular point of an object or surface in the view of camera A and camera B and the point is captured in the pixels of the two image frames. In external camera calibration, most of these points are identified and called corresponding points. Since there is a subject in the viewing field of Camera A and Camera B during the calibration, key joints of this subject are identified, eg, the center of the left wrist. If these key joints are visible in image frames from both camera A and camera B, these are assumed to represent corresponding points. This procedure is repeated for many image frames to create a large set of corresponding points for all camera pairs with overlapping viewing fields. In one embodiment, images are streamed out of all cameras at a rate of 30 FPS (frames per second) or more and a resolution of 720 pixels for full RGB (red, green, and blue) color. These images have a one-dimensional array (also known as in the form of a flat array).

The large number of images collected above for subjects can be used to determine corresponding points between cameras with overlapping viewing fields. Consider two cameras A and B with overlapping viewing fields. The plane passing through the centers of cameras A and B and joint positions (also called feature points) in the 3D scene is called the "epipolar plane". The intersection of this epipolar plane with the 2D image planes of cameras A and B defines the "epidemic". Given these corresponding points, a transformation is determined that accurately maps the corresponding point from camera A to the epipolar line in camera B's viewing field which is guaranteed to intersect the corresponding point in camera B's image frame. Using the image frame collected above for the subject, the transformation is generated. This transformation is known in the art to be nonlinear. The general form is further known to require compensation for radial deformations of the lenses of each camera, as well as non-linear coordinate transformations to and from projection space. In external camera tuning, the approximation to the ideal nonlinear transformation is determined by solving the nonlinear optimization problem. This nonlinear optimization function is used by the tracking engine 110 to identify the same joints in the outputs (arrays of joint data structures) of the different image recognition engines 112a-112n, processing images from cameras 114 with overlapping viewing fields. The results of the internal and external camera calibrations are stored in the calibration database 170 .

Various techniques may be used to determine the relative positions of points in the image of camera 114 in real space. For example, Longuet-Higgins published "Computer Algorithms for Scene Reconstruction from Two Projections" in Nature, Volume 293, September 10, 1981. This paper proposes to compute the 3D structure of the scene from the perspective projections of related pairs, when the space between the two projections is relationship is unknown. The Longuet-Higgins paper proposes a technique to determine the position of each camera in real space relative to other cameras. Furthermore, their technique allows triangulation of subjects in real space using images from cameras 114 with overlapping viewing fields to identify the value of the z-coordinate (height from the floor). Any point in real space (eg, the end of a shelf in a corner of real space) is designated as a point (0,0,0) on the (x,y,z) coordinate system of real space.

In one embodiment of the present technology, externally tuned parameters are stored in two data structures. The first data structure stores essential parameters. Essential parameters represent the projection transformation from 3D coordinates to 2D image coordinates. The first data structure contains the essential parameters of each camera, as shown below. The data values are all numeric floating point numbers. This data structure stores a 3x3 essential matrix, denoted "K" and the deformation coefficients. The deformation coefficients include six radial deformation coefficients and two tangential deformation coefficients. Radial deformation occurs when a light ray bends closer to the edge of the lens than at its optical center. Tangential deformation occurs when the lens is not parallel to the image plane. The following data structure shows the values for the first camera only. Similar data is stored for all cameras 114 .

The second data structure stores each pair of cameras: 3x3 base moments Matrix (F), 3x3 Fundamental Matrix (E), 3x4 Projection Matrix (P), 3x3 Rotation Matrix (R) and 3x1 Transform Vector (t). This data is used to convert points in one camera's reference frame to another camera's reference frame. For each pair of cameras, eight homography coefficients are also stored to map the plane of the floor 220 from one camera to the other. A fundamental matrix is a relationship between two images of the same scene that constrains where in the two images a projection from a point of the scene can occur. The fundamental matrix is also the relationship between two images of the same scene, with their cameras tuned. Projection matrices provide vector space projections from 3D real space to the subject. Rotation matrices are used to perform rotations in Euclidean space. The transformation vector "t" represents a geometric transformation, which moves each point of a figure or a space in a given direction by the same distance. The homography_floor_coefficients are used to combine images of features of the subject on the floor 220 as viewed by cameras with overlapping viewing fields. The second data structure is shown below. Similar data is stored for all pairs of cameras. As indicated previously, x represents a numeric floating point number.

network configuration

FIG. 4 presents an architecture 400 of a network that hosts an image recognition engine. The system includes a plurality of network nodes 101a-101n in the embodiment shown. In this embodiment, the network node is also referred to as a processing platform. The processing platforms 101a - 101n and cameras 412 , 414 , 416 , . . . 418 are connected to a network 481 .

Figure 4 shows a plurality of cameras 412, 414, 416, . . . 418 connected to the network. Numerous cameras can be deployed in a particular system. In one embodiment, cameras 412-418 are connected to network 481 using Ethernet-based connectors 422, 424, 426, and 428, respectively. In this embodiment, the Ethernet-based connector has a data transfer speed of gigabits per second, also known as Gigabit Ethernet. It should be understood that in other embodiments, the camera 114 is connected to the network using other types of network connections that may have faster or slower data transfer rates than Gigabit Ethernet. Also, in alternative embodiments, a set of cameras may be directly connected to each processing platform, and the processing platforms may be coupled to a network.

Storage subsystem 430 stores the basic programming and data structures that provide the functionality of certain embodiments of the present invention. For example, various modules implementing the functions of the plurality of image recognition engines may be stored in the storage subsystem 430 . The storage subsystem 430 is an example of a computer readable memory, which includes a non-transitory data storage medium having computer instructions stored in the memory that can be executed by the computer to perform the data processing and image processing described herein. All or any combination, including logic to: identify Change, track the subject, and detect the placing and removal of inventory items in the area of the real space, by the procedure as described in the text. In other examples, computer instructions may be stored in other types of memory, including portable memory, including non-transitory data storage media or media readable by a computer.

These software modules are typically executed by processor subsystem 450 . Host memory subsystem 432 typically includes several memories, including main random access memory (RAM) 434 (for storing instructions and data during program execution) and read only memory (ROM) 436 (where fixed instructions are stored) ). In one embodiment, RAM 434 is used as a buffer to store video streams from cameras 114 which are connected to platform 101a.

File storage subsystem 440 provides persistent storage for program and data files. In an example embodiment, storage subsystem 440 includes four 120-gigabyte (GB) solid-state drives (SSDs) in a RAID 0 (redundant array of independent drives) configuration (identified by numeral 442) . In an example embodiment where a CNN is used to identify joints of the subject, RAID 0 442 is used to store training data. During training, its training data not in RAM 434 is read from RAID 0 442. Similarly, when images are recorded for training purposes, their data not in RAM 434 is stored in RAID 0 442. In an example embodiment, the hard disk drive (HDD) 446 is 10 megabytes of storage. It has slower access speed than RAID 0 442 storage. Solid State Drive (SSD) 444 contains the operating system and associated files for image recognition engine 112a.

In an example configuration, three cameras 412, 414, and 416 is connected to the processing platform 101a. Each camera has dedicated graphics processing units GPU 1 462, GPU 2 464, and GPU 3 466 for processing the images transmitted by the cameras. It should be understood that less or more than three cameras may be connected to each processing platform. Therefore, fewer or more GPUs are configured in a network node so that each of its cameras has a dedicated GPU for processing image frames received from that camera. Processor subsystem 450 , storage subsystem 430 , and GPUs 462 , 464 , and 466 communicate using bus subsystem 454 .

Several peripheral devices, such as a network interface subsystem, user interface output devices, and user interface input devices, are also connected to the streaming subsystem 454, which forms part of the processing platform 101a. These subsystems and devices are intentionally not shown in FIG. 4 to improve clarity of illustration. Although the bus subsystem 454 is shown diagrammatically as a single bus, alternative embodiments of the bus subsystem may use multiple buses.

In one embodiment, the camera 412 may be implemented using a Chameleon3 1.3 MP Color USB3 Vision (Sony ICX445) with a resolution of 1288x964, a frame rate of 30 FPS, and at 1.3 megapixels per image (MegaPixels) using 300-∞ working distance (mm) zoom lens with 1/3" sensor viewing field of 98.2°-23.8°.

Convolutional Neural Network

The image recognition engine in the processing platform receives a continuous stream of images at a predetermined rate. In one embodiment, the image recognition engines include convolutional neural networks (abbreviated as CNNs).

FIG. 5 illustrates the image frame by the CNN represented by the number 500 processing. The input image 510 is a matrix of image pixels arranged in columns and rows. In one embodiment, the input image 510 has a width of 1280 pixels, a height of 720 pixels, and 3 channels of red, blue, and green (also known as RGB). The channels are imagined as three 1280x720 2D images stacked on top of each other. Therefore, the input image has dimensions of 1280x720x3 as shown in FIG. 5 .

The 2x2 filter 520 is convolved with the input image 510. In this embodiment, no padding is applied when the filter is convolved with the input. Continuing here, a nonlinear function is applied to the convoluted image. In this embodiment, Corrected Linear Unit (ReLU) activation is used. Other examples of nonlinear functions include sigmoid, hyperbolic tangent (tanh), and variations of ReLU, such as leaky ReLU. A search is performed to find hyperparameter values. The hyperparameters are C ₁ , C ₂ , ....., _CN , where _CN represents the number of channels of the convolutional layer "N". Typical values of N and C are shown in FIG. 5 . There are twenty-five (25) layers in the CNN, as represented by N equal to 25. The value of C is the number of channels in each of the convolutional layers from layers 1 to 25. In other embodiments, additional features are added to the CNN 500, such as residual connections, squeeze excitation modules, and multiple resolutions.

In a typical CNN for image classification, the size (width and height dimensions) of an image is reduced as the image is processed through convolutional layers. It is helpful for feature recognition since the goal is to predict the class of the input image. However, in the embodiment shown, the size of the input image (ie, the image width and height dimensions) is not reduced because the goal is not only to identify joints (aka features) in the image frame, but It also identifies its position in the image, so it can be mapped to a seat in real space mark. Thus, as shown in Figure 5, the width and height dimensions of the image remain unchanged as the process proceeds through the convolutional layers of the CNN, in this example.

In one embodiment, CNN 500 identifies 19 possible joints of the bodies on elements of the image. The possible joints can be clustered into two categories: foot joints and non-foot joints. Type 19 joint classes are for all non-joint features of the subject (ie, elements not classified as images of joints).

Foot joints:

Ankle (left and right)

Non-foot joints:

neck

nose

Eyes (left and right)

Ears (left and right)

Shoulders (left and right)

Elbow (left and right)

Wrist (left and right)

Buttocks (left and right)

Knee (left and right)

not a joint

As can be seen, for the purposes of this specification, a "joint" is a traceable feature of a subject in real space. The joints may correspond to physical joints on the bodies, or other features (such as eyes, or noses).

The first set of analyses on the stream of input images identifies traceable features of subjects in real space. In one embodiment, this is called "joint analysis." In this embodiment, the CNN used for joint analysis is called "joint CNN". In one embodiment, joint analysis is performed thirty times per second, on thirty frames per second received from the corresponding cameras. The analysis is time-synchronized (ie, 1/ ^30th of a second), and images from all cameras 114 are analyzed in corresponding joint CNNs to identify joints of all subjects in real space. The results of this analysis of images from multiple cameras from a single moment in time are stored as "snapshots".

A snapshot may be in the form of a dictionary of images from all cameras 114 at a time containing an array of joint data structures representing clusters of candidate joints within the region of real space covered by the system. In one embodiment, the snapshots are stored in the subject database 140 .

In this example CNN, the softmax function is applied to each element of the image in the final layer of convolutional layer 530. The softmax function transforms any real-valued K-dimensional vector into a real-valued K-dimensional vector in the range [0,1] (which adds up to 1). In one embodiment, the element of the image is a single pixel. The softmax function converts an arbitrary real-valued 19-dimensional array (also known as a 19-dimensional vector) for each pixel to a real-valued 19-dimensional confidence array in the range [0,1] (which adds up to 1). The 19 dimensions of the pixels in the image frame correspond to the 19 channels in the final layer of the CNN, which further correspond to the 19 types of joints of the subjects.

A large number of picture elements can be classified into one of 19 types of joints in an image, according to the number of subjects in the viewing field of the source camera for the image.

Image recognition engines 112a-112n process the images to generate Confidence array of elements for this image. The confidence array for a particular element of the image includes confidence values for a plurality of joint types for that particular element. Each of the image recognition engines 112a-112n (respectively) generates an output matrix 540 of confidence arrays per image. Finally, each image recognition engine generates an array of joint data structures for each output matrix 540 corresponding to the confidence array for each image. The array of joint data structures corresponding to a particular image classifies the elements of a particular image by the joint type, the time of the particular image, and the coordinates of the elements in the particular image. The joint type of the joint data structure of a particular element in each image is selected based on the values of the confidence array.

The joints of the bodies can be viewed as being distributed in the output matrix 540 as a heatmap. The heat map can be resolved to display image elements with the highest values (peaks) for each joint type. Ideally, for a given picture element with a high value for a particular joint type, surrounding picture elements that are outside a certain distance from the given picture element will have lower values for that joint type, so that they have the same The location of a particular joint can be identified in image space coordinates. Accordingly, the confidence array for the image element will have the highest confidence value for that joint and lower confidence values for the remaining 18 types of joints.

In one embodiment, batches of images from each camera 114 are processed by respective image recognition engines. For example, six consecutive time-stamped images are processed sequentially in a batch to take advantage of cache coherence. Parameters for one layer of the CNN 500 are loaded into memory and applied to the batch of six image frames. The parameters for the next layer are then loaded into memory and applied to the batch of six images. This is repeated for all convolutional layers in CNN 500 530. Cache coherence reduces processing time and improves image recognition engine performance.

In one such embodiment, for three-dimensional (3D) convolution, a further improvement in the performance of the CNN 500 is achieved by sharing information across the image frames in the batch. This assists in more accurate identification of joints and reduces false positives. For example, features in image frames in which pixel values across most image frames in a given batch do not change are likely to be static objects (such as shelves). A change in value for the same pixel across an image frame in a given batch indicates that this pixel is likely to be a joint. Therefore, the CNN 500 can focus more on processing the pixel to correctly identify its joint identified by the pixel.

joint data structure

The output of CNN 500 is a matrix of confidence arrays for each image of each camera. The matrix of confidence arrays is transformed into an array of joint data structures. The joint data structure shown in Figure 6 is used to store the information of each joint. The joint data structure 600 identifies the x and y positions of elements in a particular image in the 2D image space of the camera from which the image is received. The joint number identifies the type of joint it has identified. For example, in one embodiment, the values range from 1 to 19. A value of 1 indicates that the joint is the left ankle, a value of 2 indicates that the joint is the right ankle, and so on. The type of joint is selected using the confidence array for that element in output matrix 540 . For example, in one embodiment, if the value corresponding to the left ankle joint is the highest in the confidence array for that image element, then the value of the number of joints is "1".

The confidence number indicates the degree of confidence in the CNN 500 in predicting the joint. If the confidence number is high, it means that the CNN is confident in its prediction. An integer Id is assigned to the joint data structure to uniquely identify it. Following the above mapping, the output matrix 540 of the confidence array for each image is converted into an array of joint data structures for each image.

Image recognition engines 112a-112n receive sequences of images from camera 114 and process the images to generate corresponding arrays of joint data structures as described above. The array of joint data structures for a particular image classifies the elements of a particular image by the joint type, the time of the particular image, and the coordinates of the elements in the particular image. In one embodiment, the image recognition engines 112a-112n are convolutional neural networks CNN 500, the joint type is one of the 19 types of joints of the subjects, and the time of a specific image is determined by the source camera 114 for the specific image The timestamp of the generated image, and the coordinates (x,y) identify the location of the element on the 2D image plane.

In one embodiment, joint analysis includes performing k-nearest neighbors, a mixture of Gaussians, various image morphological transformations, and a combination of joint CNNs on each input image. The result contains an array of joint data structures, which can be stored as bit masks in the ring buffer, which map the number of images to the bit masks at each instant.

tracking engine

Tracking engine 110 is configured to receive an array of joint data structures generated by image recognition engines 112a-112n, corresponding to An image in an image sequence of cameras with overlapping viewing fields. The array of joint data structures for each image is sent by image recognition engines 112a-112n to tracking engine 110 via network 181 as shown in FIG. The tracking engine 110 transforms the coordinates of the elements in the array of joint data structures corresponding to the images in the different sequences into candidate joints with coordinates in real space. Tracking engine 110 includes logic to identify sets of candidate joints (clusters of joints) with coordinates in real space as subjects in real space. In one embodiment, tracking engine 110 accumulates an array of image recognition engine joint data structures from all cameras at a given time, and stores this information as a dictionary in master database 140 for use in identifying clusters of candidate joints . The dictionary can be configured as a key-value pair, where the key is the camera id and the value is an array of joint data structures from that camera. In this embodiment, this dictionary is used for heuristic-based analysis to determine candidate joints and assignments of joints to the subject. In this embodiment, the high-level inputs, processing, and outputs of the tracking engine 110 are set forth in Table 1.

Cluster joints into candidate joints

The tracking engine 110 receives an array of joint data structures along two dimensions: time and space. Along the time dimension, the tracking engine sequentially receives the time-stamped array of joint data structures processed by each camera's image recognition engine 112a-112n. The joint data structure includes most instances of the same joint of the same subject during a period of time in images from cameras with overlapping viewing fields. The (x,y) coordinates of elements in a particular image will typically be different in the sequential timestamp array of the joint data structure due to the movement of the body to which the particular joint belongs. For example, twenty picture elements classified as left wrist joints may appear in many sequential time-stamped images from a particular camera, each left wrist joint having its real space that may or may not change with the image the location. As a result, the twenty left wrist joint data structures 600 in the sequential timestamp array of many joint data structures may represent the same twenty joints in real space at a certain period.

Because most cameras with overlapping viewing fields cover locations in real space, the same joint may appear in more than one image of one of the cameras 114 at any given time. The cameras 114 are time-synchronized so that the tracking engine 110 receives joint data structures for a particular joint from a plurality of cameras with overlapping viewing fields, at any given time. This is the spatial dimension (the second of the two dimensions: time and space) along which the tracking engine 110 receives data in the array of joint data structures.

The tracking engine 110 uses the heuristics stored in the heuristic database 160 An initial set of heuristics to identify candidate joint data structures from an array of joint data structures. The goal is to minimize the overall measure over a period of time. The overall metric calculator 702 calculates the overall metric. The overall measure is the sum of many of the values described below. Intuitively, the value of the overall metric is minimal when the joints in the array of joint data structures received by the tracking engine 110 along the temporal and spatial dimensions are correctly assigned to the respective subjects. For example, consider an embodiment of a shopping store with consumers moving in aisles. If Consumer A's left wrist is incorrectly assigned to Consumer B, the value of the overall metric will increase. Therefore, minimizing the overall measure of each joint of each consumer is an optimization problem. One option to solve this problem is to try all possible connections of the joint. However, this can become intractable as the number of consumers increases.

A second way to address this problem is to use heuristics to reduce the possible combinations of joints that are identified as members of the set of candidate joints for a single subject. For example, the left wrist joint must not belong to a body that is spatially distant from other joints of a body due to the known physiology of the relative positions of the joints. Similarly, a left wrist joint with a small change in position from image to image is unlikely to belong to a subject with the same node at the same location from an image that is distant in time, because subjects are not expected to be able to move at very high velocities move. These initial heuristics are used to establish temporal and spatial boundaries for which clusters can be classified as candidate joints for a particular subject. Joints in a joint data structure within a particular temporal and spatial boundary are considered "candidate joints" for assignment to a set of candidate joints for subjects as they appear in real space. These candidate joints are included from the same camera Joints identified in the array of joint data structures from most images, over a period of time (temporal dimension) and across different cameras with overlapping viewing fields (spatial dimension).

Foot joints

The joints may be partitioned for the purpose of a procedure of grouping the joints into clusters (being foot and non-foot joints as shown in the list of joints above). The left and right ankle joint types in the current example are considered foot joints for the purposes of this procedure. The tracking engine 110 may begin using the foot joints to identify a set of candidate joints for a particular subject. In the shopping store embodiment, the consumer's feet are on the floor 220 as shown in FIG. 2 . The distance from camera 114 to floor 220 is known. Thus, the tracking engine 110 can assume a known depth (distance along the z-axis) when combining its joint data structures for the foot joints from an array of data joint data structures corresponding to images of cameras with overlapping viewing fields . The value depth of the foot joint is zero, that is, (x, y, 0) in the (x, y, z) coordinate system of real space. Using this information, the image tracking engine 110 applies a homography map to combine the joint data structures of the foot joints from cameras with overlapping viewing fields to identify candidate foot joints. Using this mapping, the positions of joints in (x,y) coordinates in image space are transformed to positions in (x,y,z) coordinates in real space, resulting in candidate foot joints. This procedure is performed separately to identify candidate left and right foot joints using respective joint data structures.

Continuing here, the tracking engine 110 may combine the candidate left foot joints with the candidate right foot joints (which are assigned to the set of candidate joints) to generate living subject. Other joints from the galaxy of candidate joints may be linked to the body to create clusters of some or all joint types of the resulting body.

If there is only one left candidate foot joint and one right candidate foot joint, it means that there is only one subject in the specific space at the specific time. The tracking engine 110 generates a new body with left and right candidate foot joints belonging to its joint set. The subject is stored in subject database 140 . If there are a majority of candidate left and right foot joints, the overall metric calculator 702 attempts to combine each candidate left foot joint to each candidate right foot joint to generate a body such that the value of its overall metric is minimized.

non-foot joints

To identify candidate non-foot joints from an array of joint data structures within specific temporal and spatial boundaries, the tracking engine 110 uses a nonlinear transformation (also called the fundamental matrix). Non-linear transformations are computed using a single multi-joint body and stored in the tuning database 170 as described above. For example, for two cameras A and B with overlapping viewing fields, candidate non-foot joints are identified as follows. Non-foot joints in the array of joint data structures corresponding to elements in the image frame from camera A are mapped to epipolar lines in the synchronized image frame from camera B. A joint identified by a joint data structure in the array of joint data structures for a particular image of camera A (also known as a feature in machine vision literature) would appear on the corresponding epipolar line if it appeared in the image of camera B . For example, if the joint in the joint data structure from camera A is the left wrist joint, then the The left wrist joint on the epipolar line represents the same left wrist joint from camera B's view. These two points in the images of cameras A and B are projections of the same point in the 3D scene in real space and are called "conjugate pairs".

Machine vision techniques (such as the technique published by Longuet-Higgins in a paper entitled "Computer Algorithms for Reconstructing Scenes from Two Projections", Nature, Volume 293, September 10, 1981) were applied to Conjugate pairs of corresponding points are determined to determine the height of the joint from the floor 220 in real space. Application of the above method requires a predetermined mapping between cameras with overlapping viewing fields. This data is stored in the calibration database 170 as a nonlinear function determined during the calibration of the camera 114 described above.

Tracking engine 110 receives an array of joint data structures corresponding to images in a sequence of images from cameras having overlapping viewing fields, and transforms the coordinates of elements in the array of joint data structures corresponding to images in different sequences to have Candidate non-foot joints for coordinates in real space. The identified candidate non-foot joints are clustered using population metric calculator 702 as a set of subjects with coordinates in real space. The global metric calculator 702 calculates a global metric value and attempts to minimize this value by examining different combinations of non-foot joints. In one embodiment, the overall metric is the sum of the heuristics organized in four categories. The logic for identifying the set of candidate joints includes a heuristic function for identifying the set of candidate joints as a subject based on the physical relationship between the subject's joints in real space. Examples of physical relationships between joints are considered in the heuristics described below.

first kind of heuristics

A first class of heuristics includes metrics to determine the similarity between two proposed body-joint positions in the same camera view at the same or different time instants. In one embodiment, these metrics are floating point values, where a higher value indicates that the two lists of joints are most likely to belong to the same subject. Considering the example embodiment of a shopping store, the metrics determine the distance between the same joints of the consumer in a camera from one image to the next along the time dimension. Given camera A in the viewing field of camera 412, a first set of metrics determines the distance between each of the joints of person A from one image from camera 412 to the next image from camera 412. These metrics are applied to the joint data structure 600 in the array of joint data structures per image from the camera 114 .

In one embodiment, two example metrics of the first type of heuristic are listed below:

1. The inverse of the Euclidean 2D coordinate distance between the left ankle joints of the two subjects on the floor and the right ankle joints of the two subjects on the floor (using x,y for a specific image from a specific camera coordinate values) are added together.

2. The sum of the Euclidean 2D coordinate distances between each pair of non-foot joints of the subject in the image frame.

The second kind of heuristic

A second type of heuristic involves metrics, which are used to determine Similarity between two proposed subject-joint positions from the viewing field of most cameras at the same time. In one embodiment, these metrics are floating point values, where a higher value indicates that the two lists of joints are most likely to belong to the same subject. Considering the example embodiment of a shopping store, a second set of metrics determines the distance between the same joints of the consumer between image frames from two or more cameras (with overlapping viewing fields) at the same time.

In one embodiment, two example metrics of the second type of heuristic are listed below:

1. The inverse of the Euclidean 2D coordinate distance between the left ankle joints of the two subjects on the floor and the right ankle joints of the two subjects on the floor (using x,y for a specific image from a specific camera coordinate values) are added together. The ankle joint position of the first subject is projected to the camera, where the second subject is visible through the homography mapping.

2. The sum of all pairs of joints of the reciprocal Euclidean 2D coordinates between a line and a point, where the line is from the first camera with the first subject in its viewing field to the second subject in it The epipolar line of the joint in the image of the second camera in the viewing field, and this point is the joint of the second body in the image from the second camera.

The third kind of heuristic

A third class of heuristics includes metrics to determine the similarity between all joints at the same time and in the same camera view at the proposed body-joint position. Considering the example embodiment of a shopping store, such The class metric determines the distance between the consumer's joints in a frame from a camera.

The fourth kind of heuristic

A fourth class of heuristics includes metrics to determine the degree of dissimilarity between proposed body-joint positions. In one embodiment, these metrics are floating point values. A higher value indicates that the two lists of joints are more likely not to be the same body. In one embodiment, two example metrics in this category include:

1. The distance between the neck joints of the two proposed subjects.

2. The sum of the distances between pairs of joints between two bodies.

In one embodiment, various threshold values, which can be determined empirically, are applied to the metrics listed above as follows:

1. Threshold value to determine when the metric is small enough to consider one of its joints belonging to a known subject.

2. Threshold value to determine when there are too many potential candidates, one of which may belong to a metric similarity score that is too good.

3. Threshold value to determine when a set of joints has a high enough metric similarity to be considered a new subject, not previously present in real space.

4. Threshold value to determine when the subject is no longer in the real space.

5. Threshold value to determine when the tracking engine 110 has generated an error and confused the two agents.

Tracking engine 110 includes logic to store the set of joints it identifies as a subject. The logic to identify the set of candidate joints includes logic to determine whether a candidate joint identified in an image taken at a particular time is a member of one of the sets of candidate joints that were identified as a subject in a previous image . In one embodiment, the tracking engine 110 compares a subject's current joint positions with previously recorded joint positions for the same subject at regular intervals. This comparison allows the tracking engine 110 to update the body's joint positions in the real space. Furthermore, using this approach, the tracking engine 110 identifies false positives (ie, misidentified subjects) and removes subjects that no longer appear in the real space.

Consider an example of a shopping store embodiment where the tracking engine 110 generates a customer (subject) at an earlier time, however, after a certain time, the tracking engine 110 does not have the current joint position for that particular customer. It indicates that the consumer was generated incorrectly. Tracking engine 110 deletes incorrectly generated subjects from subject database 140 . In one embodiment, the tracking engine 110 also uses the above procedure to remove positively identified subjects from real space. Considering the shopping store example, when a consumer leaves the shopping store, the tracking engine 110 deletes the corresponding consumer record from the subject database 140 . In one such embodiment, the tracking engine 110 updates the customer's record in the subject database 140 to indicate that "the customer has left the store."

In one embodiment, the tracking engine 110 attempts to identify the subject by applying foot and non-foot heuristics simultaneously. This results in "islands" of the connecting joints of the bodies. The size of the islands increases as the tracking engine 110 processes further arrays of joint data structures along the temporal and spatial dimensions. Finally, islands of joints are merged into other islands of joints to form bodies, which are then stored in body database 140 . In one embodiment, the tracking engine 110 maintains a record of unassigned joints for a predetermined period of time. During this time, the tracking engine attempts to assign unassigned joints to existing bodies or to generate new multi-joint monomers from these unassigned joints. The tracking engine 110 discards the unassigned joints after a predetermined period of time. It should be understood that in other embodiments, different heuristics than those listed above are used to identify and track subjects.

In one embodiment, a user interface output device connected to node 102 of master tracking engine 110 displays the position of each subject in the real space. In one such embodiment, the display of the output device is refreshed at regular intervals with the new positions of the subjects.

main data structure

The joints of the bodies are connected to each other using the metrics described above. In doing so, the tracking engine 110 generates new bodies and updates the positions of existing bodies by updating their respective joint positions. FIG. 8 shows a subject data structure 800 for storing subjects. Data structure 800 stores subject-related data as a key-value dictionary. The key is the box_number and the value is another key-value dictionary, where the key is the camera_id and the value is a list of 18 joints (of the subject) with their positions in real space. The subject data is stored in subject database 140 . Each new subject is also assigned a unique identifier, which is used to access that subject's data in subject database 140 .

In one embodiment, the system identifies the joints of the subject and generates The skeleton of the subject is born. The bone is projected into real space to indicate the position and orientation of the subject in real space. This is also known as "pose estimation" in the field of machine vision. In one embodiment, the system displays the orientation and position of the subject in real space on a graphical user interface (GUI). In one embodiment, the image analysis is anonymous, that is, the unique identifiers assigned to subjects generated through joint analysis do not identify personally identifiable details (such as names, email addresses, etc.) of any particular subject in real space. , address, credit card number, bank account number, driver's license number, etc.).

Program flow of subject tracing

Several flowcharts illustrating the logic are described in the text. Logic can be implemented: using processors configured as described above, programmed using a computer program stored in memory accessible and executable by the processors; and in its configuration, By means of dedicated logic hardware (including field programmable integrated circuits), and by a combination of dedicated logic hardware and computer programs. Using all flowcharts herein, it should be understood that many of the steps may be combined, performed in parallel, or performed in different sequences without affecting the functionality achieved. In some cases, as the reader will understand, a reconfiguration of steps will achieve the same result, only if some other change is also performed concurrently. In other cases, as the reader will understand, reconfiguration of steps will achieve the same result, only if certain conditions are met. Furthermore, it should be understood that the flowcharts herein only show steps which are relevant to an understanding of the embodiments, and that it should be understood that various other steps to accomplish other functions may be performed before, after, and between those shown.

FIG. 9 is a flowchart illustrating the procedural steps for tracking a subject. The procedure begins at step 902. The camera 114 with the viewing field of the region in real space is calibrated at program step 904 . The video process is performed at step 906 by the image recognition engines 112a-112n. In one embodiment, a video program is executed per camera to process batches of image frames received from respective cameras. The output of all video programs from the respective image recognition engines 112a - 112n is provided as input to the scene program executed by the tracking engine 110 at step 908 . The scene program recognizes the new body and updates the joint positions of the existing body. At step 910, it is checked whether there are more frames to be processed. If there are more image frames, the process continues at step 906 , otherwise the process ends at step 914 .

More detailed procedural steps of procedural step 904 "Calibrate the camera in real space" are presented in the flowchart of FIG. 10 . The calibration procedure begins at step 1002 by identifying the (0,0,0) point of the (x,y,z) coordinate in real space. At step 1004, a first camera with position (0,0,0) in its viewing field is calibrated. Further details of camera calibration were presented earlier in this application. At step 1006, the next camera with an overlapping viewing field with the first camera is calibrated. In step 1008, it is checked whether there are more cameras to be adjusted. This procedure is repeated at step 1006 until all cameras 114 have been calibrated.

In the next procedural step 1010, the subject is introduced into real space to identify conjugate pairs of corresponding points between cameras with overlapping viewing fields. Some details of this procedure are described above. The process is repeated at step 1012 for each pair of overlapping cameras. If there are no more cameras then the routine ends (step 1014).

The flow chart in FIG. 11 shows the more detailed steps of the "Video Procedure" step 906 . At step 1102, k consecutive time-stamped images per camera are selected as a batch for further processing. In one embodiment, the value of k=6 is calculated based on the available memory of the video programs in the network nodes 101a-101n, which respectively host the image recognition engines 112a-112n . In the next step 1104, the size of the image is set to the appropriate size. In one embodiment, the image has a width of 1280 pixels, a height of 702 pixels, and three channels RGB (representing red, green, and blue). At step 1106, a complex trained convolutional neural network (CNN) processes the images and generates an array of joint data structures per image. The output of the CNN is an array of joint data structures per image (step 1108). This output is sent to the scene program, at step 1110.

FIG. 12A is a flowchart showing the first part of the more detailed steps of the "Scenario Procedure" step 908 of FIG. 9 . The scene program is combined with output from most video programs, at step 1202 . At step 1204, checking the joint data structure identifies foot joints or non-foot joints. If the joint data structure belongs to a foot joint, then a homography map is applied to combine the joint data structure corresponding to images from cameras with overlapping viewing fields, at step 1206 . This procedure identifies candidate foot joints (left and right foot joints). At step 1208, a heuristic is applied to the candidate foot joints identified in step 1206 to identify the set of candidate foot joints as subjects. At step 1210 it is checked whether the set of candidate foot joints belongs to an existing subject. If not, a new subject is created at step 1212. Otherwise, the existing principal is updated at step 1214.

Flowchart 12B shows the second part of the more detailed steps of "Scenario Procedure" step 908 . At step 1240, the data structures of the non-foot joints are combined from a plurality of arrays of joint data structures corresponding to images in a sequence of images from cameras with overlapping viewing fields. This is performed by mapping corresponding points from the first image from the first camera to the second image from the second camera with overlapping viewing fields. Some details of this procedure are described above. At step 1242, heuristics are applied to the candidate non-foot joints. At step 1246, it is determined whether the candidate non-foot joint belongs to an existing subject. If so, the existing principal is updated at step 1248. Otherwise, the candidate non-foot joint is processed again at step 1250 (after a predetermined period of time) to match it with an existing subject. At step 1252, it is checked whether the non-foot joint belongs to an existing body. If so, the body is updated at step 1256. Otherwise, the joint is discarded at step 1254.

In an exemplary embodiment, the procedures to identify new subjects, track subjects, and remove subjects that have left real space or were incorrectly generated are implemented as performed by a runtime system (also known as an inference system). Part of the "Monomer Cohesion Algorithm". A single body is a cluster of joints referred to above as a body. Cell cohesion algorithms identify cells in real space and update the positions of joints in real space to track cell movement.

FIG. 14 presents an illustration of a video program 1411 and a scene program 1415. In the embodiment shown, four video programs are displayed, each processing images from one or more cameras 114 . The video program processes the images as described above and identifies the joints of each frame. In one embodiment, each video program identifies 2D coordinates, confidence numbers, joint numbers, and unique IDs, for each frame per joint. all The output 1452 of the video program is provided as input 1453 to the scene program 1415. In one embodiment, the scene program generates a per-time joint key-value dictionary, where the key is a camera identifier and the value is an array of joints. The joints are reprojected into the view of the cameras with overlapping viewing fields. The reprojected joints are stored as key-value dictionaries and can be used to generate foreground subject masks for each image in each camera, as discussed below. The key in this dictionary is the combination of the joint id and the camera id. The values in this dictionary are the 2D coordinates of the joints whose viewpoints are reprojected into the target camera.

The scene program 1415 produces an output 1457 that contains a list of all subjects in real space at a certain moment in time. The list includes a key-value dictionary for each subject. The key is the unique identifier of the subject and the value is another key-value dictionary with the key as the frame number and the value as the camera-subject joint key-value dictionary. The camera-body joint key-value dictionary is a per-body dictionary, where the key is the camera identifier and the value is a list of joints.

Image analysis to identify and track inventory items per entity

A system and various embodiments for tracking the placing and taking of inventory items by subject in the area of real space are described with reference to FIGS. 15A-25 . The system and process are described with reference to FIG. 15A, an architectural level overview of the system according to an embodiment. Since FIG. 15A is an architectural diagram, some details are omitted to improve the clarity of the description.

Architecture of Multiple CNN Pipelines

Figure 15A is a pipeline of a convolutional neural network (also known as a multi-CNN Pipeline) high-level architecture that processes image frames received from camera 114 to generate a shopping cart data structure for subjects in real space. The system described herein includes a per-camera image recognition engine as described above for identifying and tracking multi-joint subjects. Alternative image recognition engines may be used, including instances in which only one "joint" is identified and tracked per body, or other features covering space and time or other types of image data are utilized to identify and track its processed Subjects in real space.

Multiple CNN pipelines run in parallel on each camera, moving images from the respective camera to the image recognition engines 112a-112n via the circular buffer 1502 per camera. In one embodiment, the system consists of three subsystems: a first image processor subsystem 2602 , a second image processor subsystem 2604 and a third image processor subsystem 2606 . In one embodiment, the first image processor subsystem 2602 includes image recognition engines 112a-112n, which are implemented as convolutional neural networks (CNNs) and are referred to as joint CNNs 112a-112n. As described in relation to FIG. 1 , the cameras 114 may be synchronized with each other in time such that their images are captured simultaneously (or close in time) and at the same image capture rate. The images captured in all cameras which cover the area of real space simultaneously (or temporally close) are synchronized, since their synchronized images can be identified in the processing engine as represented by the fixed in real space Different perspectives of a moment in time on the subject of the location.

In one embodiment, cameras 114 are installed in a shopping store (such as a supermarket) such that a set of cameras (two or more) with overlapping viewing fields are placed over each aisle to capture the real space in the store image. There are N cameras in real space, however, for simplicity, only one camera is shown in FIG. 17A as camera( i ), where the value of i ranges from 1 to N. Each camera produces a sequence of images corresponding to the real space of its respective viewing field.

In one embodiment, image frames corresponding to the sequence of images from each camera are transmitted to the respective image recognition engines 112a-112n at a rate of 30 frames per second (fps). Each image frame has a timestamp, an identification of the camera (abbreviated as "camera_id"), and a frame identification (abbreviated as "frame_id"), along with image data. Image frames are stored in a circular buffer 1502 (also referred to as a ring buffer) per camera 114 . The circular buffer 1502 stores a collection of consecutive time-stamped image frames from the respective cameras 114.

The joint CNN processes the sequence of image frames per camera and identifies 18 different types of joints for each subject appearing in its respective viewing field. The outputs of the joint CNNs 112a-112n corresponding to cameras with overlapping viewing fields are combined to map the positions of the joints from the 2D image coordinates of each camera to the 3D coordinates in real space. The joint data structure 800 of each subject (j), where j equals 1 to x, identifies the positions of the joints of subject (j) in real space. Details of the master data structure 800 are presented in FIG. 8 . In an exemplary embodiment, the joint data structure 800 is a second-order key-value dictionary for the joints of each subject. The first key is the frame_number and the value is the second key-value dictionary, with the key as the camera_id and the value as a list of joints assigned to the subject.

A data set containing the subject identified by the joint data structure 800 and the corresponding image frame from the sequence of image frames per camera is provided as input to the bounding box generator 1504 in the third image processor subsystem 2606 . The third image processor subsystem further includes a foreground image recognition engine engine. In one embodiment, the foreground image recognition engine semantically identifies important items in the foreground (ie, the shopper, their hands, and inventory items) as it relates to, for example, images from various cameras over time. Drop-off and removal of inventory items that have been passed. In the example implementation shown in FIG. 15A, the foreground image recognition engines are implemented as WhatCNN 1506 and WhenCNN 1508. The bounding box generator 1504 implements logic to process the data set to specify its bounding box that includes the image of the identified subject's hand in the images in the sequence of images. The bounding box generator 1504 identifies the positions of the hand joints in the respective source image frames of each camera, using the positions of the hand joints in the multi-joint data structure 800 corresponding to the respective source image frames. In one embodiment, where the coordinates of the joints in the host data structure indicate the positions of the joints in 3D real space coordinates, the bounding box generator maps the joint positions from the 3D real space coordinates to the images of the respective source images 2D coordinates in the box.

The bounding box generator 1504 generates bounding boxes for the hand joints in the image frame in the circular buffer of each camera 114 . In one embodiment, the bounding box is a portion of 128 pixels (width) x 128 pixels (height) of the image frame, and the hand joint is located at the center of the bounding box. In other embodiments, the size of the bounding box is 64 pixels x 64 pixels or 32 pixels x 32 pixels. For m subjects in the image frame from the camera, there can be up to 2m hand joints, and thus 2m bounding boxes. However, there are actually less than 2m hands visible in the image frame due to obstruction by other subjects or other objects. In an exemplary embodiment, the subject's hand position is inferred from the position of the elbow and wrist joints. For example, the subject's right hand position is extrapolated to use the right elbow (recognition The positions of the right wrist (recognized as p1) and the right wrist (identified as p2) are extrapolated_quantity*(p2-p1)+p2, where extrapolated_quantity is equal to 0.4. In another embodiment, the joint CNNs 112a-112n are trained using left and right hand images. Therefore, in this embodiment, the joint CNNs 112a-112n directly identify the positions of the hand joints in the image frame of each camera. The hand position per image frame is used by the bounding box generator 1504 to generate a bounding box for each identified hand joint.

WhatCNN 1506 is a convolutional neural network trained to process specified bounding boxes in the imagery to generate classes of identified subjects' hands. A trained WhatCNN 1506 processes image frames from a camera. In the example embodiment of the shopping store, for each hand joint in each image frame, WhatCNN 1506 identifies whether the hand joint is empty. WhatCNN 1506 also identifies the SKU (Stock Keeping Unit) number of an item in stock in the hand joint, a confidence value indicating that the item in the hand joint is a non-SKU item (ie, it does not belong to shopping store inventory), and the hand in the image frame Background for joint positions.

The output of the WhatCNN model 1506 for all cameras 114 is processed by a single WhenCNN model 1508 for a predetermined time window. In the shopping store example, WhenCNN 1508 performs a time series analysis on both hands of the subject to identify whether the subject removes store inventory items from the shelves or places store inventory items on the shelves. A shopping cart data structure 1510 (also referred to as a log data structure that includes a list of inventory items) is generated for each subject to maintain a record of the store's inventory items in the shopping cart (or basket) associated with that subject.

The second image processor subsystem 2604 receives the same data set For a given input to the third image processor, these same data sets include the subject identified by the joint data structure 800 and the corresponding image frame from the sequence of image frames for each camera. Subsystem 2604 includes a foreground image recognition engine that semantically identifies significant differences in the foreground (ie, inventory display structures such as shelves) as it relates to, for example, imagery from various cameras over time. Putting down and taking out inventory items. The selection logic (not shown in FIG. 15A ) uses the confidence score to select the output from either the second image processor or the third image processor to generate the shopping cart data structure 1510 .

Figure 15B shows a coordination logic module 1522 that combines the results of multiple WhatCNN models and provides as input to a single WhenCNN model. As described above, two or more cameras with overlapping viewing fields capture images of subjects in real space. The joints of a single subject may appear in the image frames of most cameras in respective image channels 1520. A separate WhatCNN model identifies the SKUs of inventory items in the subject's hand (represented by the hand joint). The coordination logic module 1522 combines the output of the WhatCNN model into a single merged input of the WhenCNN model. WhenCNN model 1508 operates on the combined input to generate the subject's shopping cart.

Detailed implementations of the system including the multiple CNN pipelines of FIG. 15A are presented in FIGS. 16 , 17 , and 18 . In the example of a shopping store, the system tracks the placing and taking of inventory items by subject in the area of the real space. The area of the real space is a shopping store with inventory items placed in shelves organized in aisles as shown in FIGS. 2 and 3 . It should be understood that racks containing inventory items can be organized in a number of different configurations. E.g, The shelves can be arranged in a straight line with their back sides against the wall of the shopping store and their front sides towards the open area in the real space. A plurality of cameras 114 with overlapping viewing fields in real space generate a sequence of images of their corresponding viewing fields. The viewing field of a camera overlaps the viewing field of at least one other camera as shown in FIGS. 2 and 3 .

Articulation CNN-Subject Recognition and Update

16 is a flowchart of the processing steps performed by joint CNNs 112a-112n to identify subjects in real space. In the shopping store example, the subjects are consumers in the store moving in the aisles between the shelves and other open spaces. The process begins at step 1602. NOTE: As mentioned above, the camera is calibrated before the sequence of images from the camera is processed to identify the subject. Details of camera calibration are presented above. Cameras 114 with overlapping viewing fields capture images of the real space in which the subject appears (step 1604). In one embodiment, the cameras are configured to generate a synchronized sequence of images. The sequence of images for each camera is stored in a respective circular buffer 1502 for each camera. A circular buffer (also known as a ring buffer) stores a sequence of images in a sliding window of time. In one embodiment, the circular buffer stores 110 image frames from corresponding cameras. In another embodiment, each circular buffer 1502 stores image frames for a time period of 3.5 seconds. It should be understood that in other embodiments, the number of image frames (or time periods) may be greater or less than the exemplary values listed above.

The joint CNNs 112a-112n receive a sequence of image frames from the respective cameras 114 (step 1606). Each joint CNN is passed through the majority of convolution The network layer processes the batches of images from the corresponding cameras to identify the joints of the subject in the image frames from the corresponding cameras. The architecture and processing of images by an example convolutional neural network is presented in FIG. 5 . Because the cameras 114 have overlapping viewing domains, the joints of the subject are identified by more than one joint CNN. Two-dimensional (2D) coordinates of the joint data structure 600 generated by the joint CNN are mapped to three-dimensional (3D) coordinates in real space to identify joint positions in real space. Details of this mapping are presented in the discussion of Figure 7, where the tracking engine 110 transforms the coordinates of elements in the array of joint data structures corresponding to images in different image sequences into candidate joints with coordinates in real space.

The joints of the subject are organized into two classes (foot joints and non-foot joints) to group these joints into clusters, as discussed above. The left and right ankle joint types in the current example are considered foot joints for the purposes of this procedure. At step 1608, heuristics are applied to assign candidate left foot joints and candidate right foot joints to the set of candidate joints to generate the body. Continuing from this, in step 1610, it is determined whether the newly identified subject already exists in the real space. If not, a new principal is created at step 1614, otherwise, the existing principal is updated at step 1612.

Other joints from the galaxy of candidate joints may be linked to the body to create clusters of some or all joint types of the resulting body. At step 1616, heuristics are applied to the non-foot joints to assign those to the identified subject. The global metric calculator 702 calculates a global metric value and attempts to minimize this value by examining different combinations of non-foot joints. In one embodiment, the overall metric is a heuristic organized in four categories sum, as above.

The logic for identifying the set of candidate joints includes a heuristic function for identifying the set of candidate joints as a subject based on the physical relationship between the subject's joints in real space. At step 1618, the existing host system is updated using the corresponding non-foot joints. If there are more images to process (step 1620), steps 1606-1618 are repeated, otherwise the process ends at step 1622. The first data set is generated at the end of the above procedure. The first dataset is an identified subject and the location of the identified subject in real space. In one embodiment, the first data set is presented above with respect to Figure 15A as a per-subject joint data structure 800.

WhatCNN-Classification of Hand Joints

Figure 17 is a flow diagram illustrating the processing steps for identifying an inventory item in the hands of an identified subject in real space. In the shopping store example, the subject is the consumer in the shopping store. As consumers move through aisles and open spaces, they pick up inventory items stacked on shelves and place those items into their shopping carts or baskets. The image recognition engine identifies the subject in the set of images in the sequence of images it receives from the plurality of cameras. The system includes logic to process the set of images in the sequences including images of the identified subject to detect the taking of inventory items by the identified subject and the placing of the inventory items on the shelf by the identified subject.

In one embodiment, the logic for processing the set of images includes (for the identified subject) logic for processing the images to generate a class of images of the identified subject. These categories include whether the identified subject holds In stock items. The categories include a first proximity category that indicates the position of the identified subject's hand relative to the shelf. The categories include a second proximity category that indicates the position of the identified subject's hand relative to the identified subject's body. The categories further include a third proximity category that indicates the position of the identified subject's hand relative to the basket associated with the identified subject. Finally, these categories include identifiers for possible inventory items.

In another embodiment, the logic for processing the set of images includes (for identified subjects) logic to identify the delimitation of data representing hands in images in the set of images of the identified subjects frame. The data in the bounding boxes are processed to generate categories for the data in the bounding boxes for the identified subjects. In this embodiment, the category identifies whether the identified entity holds inventory items. The categories include a first proximity category that indicates the position of the identified subject's hand relative to the shelf. The categories include a second proximity category that indicates the position of the identified subject's hand relative to the identified subject's body. The categories include a third proximity category that indicates the position of the identified subject's hand relative to the basket associated with the identified subject. Finally, these categories include identifiers for possible inventory items.

The process begins at step 1702. At step 1704, the position of the subject's hand (represented by the hand joint) in the image frame is identified. Bounding box generator 1504 identifies the subject's hand positions for each frame from each camera, using the joint positions identified in the first dataset generated by joint CNNs 112a-112n as described in FIG. 18 . Continuing here, at step 1706, the bounding box generator 1504 processes the data set to specify that it includes images in the sequence of images The bounding box of the image of the identified multi-joint subject's hand in the image. Details of the bounding box generator are presented in the discussion of Figure 15A above.

A second image recognition engine receives the sequence of images from the plurality of cameras and processes specified bounding boxes in the images to generate a class of the identified subject's hand (step 1708). In one embodiment, each of the image recognition engines used to classify the subjects from hand images includes a trained convolutional neural network, referred to as WhatCNN 1506. WhatCNN is configured in a multi-CNN pipeline, as described above in relation to Figure 15A. In one embodiment, the input to WhatCNNj is a multidimensional array BxWxHxC (also known as a BxWxHxC tensor). "B" is the batch size, which indicates the number of image frames in a batch of images processed by WhatCNN. "W" and "H" indicate the width and height of the bounding box in pixels, and "C" is the number of channels. In one embodiment, there are 30 images in a batch (B=30), so the size of the bounding box is 32 pixels (width) x 32 pixels (height). There can be six channels, which respectively represent red, green, blue, foreground mask, forearm mask and upper arm mask. The foreground mask, the forearm mask, and the upper arm mask are additional and optional input data sources for WhatCNN in this example, which are information that the CNN can include in processing to classify the RGB image data. The foreground mask can be generated using, for example, a mixture of Gaussian algorithms. The forearm mask can be the line between the wrist and the elbow that provides the background generated using the information in the joint data structure. Likewise, the upper arm mask can be the line between the elbow and the shoulder, which is generated using information in the joint data structure. Different values for the B, W, H and C parameters may be used in other embodiments. For example, in another embodiment, the bounding box is The size is larger, for example, 64px (width) x 64px (height) or 128px (width) x 128px (height).

Each WhatCNN 1506 processes batches of images to generate the categories of the identified subject's hands. These categories include whether the identified entity holds inventory items. The categories include one or more categories that indicate the position of the hands relative to the shelf and relative to the body, undetectable for dropping and removing. In this example, the first proximity category indicates the position of the identified subject's hand relative to the shelf. The categories include, in this example, a second proximity category that indicates the position of the identified subject's hand relative to the identified subject's body, where the subject may be holding inventory items during shopping. The categories further include, in this example, a third proximity category that indicates the position of the identified subject's hand relative to the basket associated with the identified subject, where "basket" in this context is used by the subject for Bags, baskets, carts or other items that hold inventory items during shopping. Finally, these categories include identifiers for possible inventory items. The last layer of WhatCNN 1506 produces logits, which are the raw values of the prediction. Logits are represented as floating point values and further processed (as described below) for producing classification results. In one embodiment, the output of the WhatCNN model includes a multidimensional array BxL (also known as a BxL tensor). "B" is the batch size, and "L=N+5" is the number of logit outputs per frame. "N" is the number of SKUs representing "N" unique inventory items available for sale in the shopping store.

The output "L" for each frame is the original activation from WhatCNN 1506. Logit "L" is processed at step 1710 to identify inventory project and background. The first "N" logs represent its confidence that the entity is holding one of the "N" inventory items. A Logit "L" includes an additional five (5) Logits, which are explained below. First Logit represents its confidence that the image of the item in the subject's hands is not one of the store SKU items (also known as a non-SKU item). The second logit indicates whether the entity holds confidence in the project. A large positive value indicates that the WhatCNN model has a high level of confidence that the subject is holding the item. A large negative value indicates that the model is confident that the subject does not hold any items. A value close to zero for the second logit indicates that the WhatCNN model has no confidence in predicting whether the subject holds an item.

The next three logs represent first, second, and third proximity categories, including: a first proximity category, which indicates the position of the identified subject's hand relative to the shelf, a second proximity category, which is indicative of the position of the identified subject's hand relative to the identified subject's body, and a third proximity category is indicative of the position of the identified subject's hand relative to the basket associated with the identified subject. Thus, the three logit representations have a background with a logit's hand position, each indicating the confidence that the background of its hand is close to the shelf, close to the basket (or shopping cart), or close to the subject's body. In one embodiment, WhatCNN is trained using a training dataset containing hand images in three contexts: close to the shelf, close to the basket (or shopping cart), and close to the subject's body. In another embodiment, the "proximity" parameter is used by the system to classify the context of the hand. In this embodiment, the system determines the distance of the identified subject's hand to the shelf, basket (or shopping cart), and the subject's body to classify the context.

The output of WhatCNN is "L" Logit, which includes: N SKU LOGIT, 1 non-SKU LOGIT, 1 HOLD LOGIT, and 3 background LOGIT, as described above. SKU logs (the first N logs) and non-SKU logs (the first logs following the N logs) are handled by the softmax function. As described above with reference to FIG. 5, the softmax function transforms any real-valued K-dimensional vector to a real-valued K-dimensional vector in the range [0,1] (which adds up to 1). The softmax function computes the probability distribution of items that cover N+1 items. The output value is between 0 and 1, and the sum of all probabilities equals one. The softmax function (for multi-class classification) returns the probability of each class. The class with the highest probability is the predicted class (also called the target class).

Holding Logit is handled by a sigmoid function. The sigmoid function has real-valued inputs and produces output values in the range 0 to 1. The output of the sigmoid function identifies whether the hand is empty or holds an item. The three background logs are processed by the softmax function to identify the background of the hand joint positions. At step 1712, it is checked whether there are more images to be processed. If so, steps 1704-1710 are repeated, otherwise the process ends at step 1714.

WhenCNN-Time Series Analysis to Identify Items Putting Down and Picking Up

In one embodiment, the system implements logic to perform a time-series analysis covering classes of subjects to detect removals and placements by the identified subjects based on prior image processing of the subjects. Time series analysis identifies the gestures of the subjects and inventory items associated with the gestures represented in the sequence of images.

The output of WhatCNN 1506 in the multi-CNN pipeline is provided as input to WhenCNN 1508, which processes these inputs to detect By taking and putting down the identified subjects. Finally, the system includes logic (in response to detected picks and drops) to generate a log data structure that includes a list of inventory items for each identified subject. In the shopping store example, the log data structure is also referred to as the per-subject shopping cart data structure 1510.

Figure 18 presents a process that implements logic to generate a per-subject shopping cart data structure. The procedure begins at step 1802. The input to WhenCNN 1508 is prepared at step 1804. The input to the WhenCNN is the multidimensional array BxCxTxCams, where B is the batch size, C is the number of video channels, T is the number of boxes considered for a time window, and Cams is the number of cameras 114. In one embodiment, the batch size "B" is 64 and the value of "T" is 110 image frames or the number of image frames in a 3.5 second period.

A list of 10 LOGITS per hand joint (20 LOGITS for both hands) is generated for each subject identified per image frame, per camera. Possession and Background Logit is part of the "L" Logit produced by WhatCNN 1506, as described above.

The above data structure is generated for each hand in the image frame and also includes the other hand about the same subject. For example, if the data is for the subject's left hand joint, the corresponding value for the right hand is included as the "other" logit. The fifth logit (item number 3 in the above list called log_sku) is the logarithm of the SKU logit in the above "L" logit. The sixth Logit is the log of the other SKU Logit. The "roll" function generates the same information before and after the current frame. For example, the seventh logit (called roll(log_sku,-30)) is the logarithm of the SKU logit and is 30 frames earlier than the current frame. Eighth Logit is the log of the SKU Logit for that hand, 30 frames later than the current frame. The ninth and tenth data in this list are other similar data, 30 frames earlier and 30 frames later than the current frame. A similar data structure was also generated for the other hand, resulting in a total of 20 logs per subject per frame per camera. Therefore, the number of channels in the input to WhenCNN is 20 (ie, C=20 in the multidimensional array BxCxTxCams).

For all image frames in the batch of image frames from each camera (eg, B=64), a similar data structure of 20 hand logs per subject (identified in that image frame) is generated. A time window (T=3.5 seconds or 110 image frames) was used to search forward and backward for image frames in the sequence of image frames for the subject's hand joint. At step 1806, the 20 hand logs per subject per frame are merged from the multi-CNN pipeline. In one embodiment, the batch of image frames (64) can be thought of as the smaller windows of the image frames, which are placed in the middle of the larger windows of the image frame 110, with additional image frames for viewing on both sides. Search forward and backward. The input BxCxTxCams to WhenCNN 1508 consists of the following: A batch of image frames from all cameras 114 (referred to as "Cams") 20 logits of both hands of the subject identified in sub "B". The combined input is provided to a single trained convolutional neural network (referred to as the WhenCNN model 1508).

The output of the WhenCNN model contains 3 logs, which represent the confidence in the three possible actions of the identified subject: remove the inventory item from the shelf, put the inventory item back on the shelf, and no action. The three output logs are processed by the softmax function to predict the actions performed. Three category logs are generated at regular intervals for each subject, and the results are stored (with timestamps) for each individual. In one embodiment, three logs are generated every twenty frames per body. In this embodiment, at intervals of every twenty image frames per subject, 110 image frame windows are formed around the current image frame.

A time series analysis of these three logs per subject during a period of time is performed (step 1808) to identify gestures corresponding to real events and the times of their occurrence. A non-maximum suppression (NMS) algorithm is used for this purpose. When an event is detected by WhenCNN 1508 multiple times (from the same camera and from multiple cameras) (ie, by dropping and taking items of a subject), the NMS removes redundant events for a subject. NMS is a re-scoring technique that includes two main tasks: "matching loss" that penalizes redundant detections, and "joint processing" of neighbors to know if there are better detections nearby.

The ground truth event for each subject's take and drop is further processed by calculating the average of the SKU logit for the 30 image frames preceding the image frame with the ground truth. Finally, the maximum value of Arguments (abbreviated as arg max or argmax) are used to determine the maximum value. Inventory items sorted by the argmax value are used to identify inventory items from shelves to be put down or removed. Inventory items are added to the logarithm of the respective subject's SKU (also known as a shopping cart or basket), at step 1810. Process steps 1804 to 1810 are repeated if there are more class data (checked at step 1812). During a period of time, this process results in updates to each subject's shopping cart or basket. The process ends at step 1814.

WhatCNN with scenes and video programs

Figure 19 presents an embodiment of a system in which data from scene program 1415 and video program 1411 are provided as input to WhatCNN model 1506 to generate hand image categories. Note: The output of each video program is provided to a separate WhatCNN model. The output from scene program 1415 is the joint dictionary. In this dictionary, the key is the unique joint identifier and the value is the unique body identifier that the joint is associated with. If no body is associated with the joint, it is not included in the dictionary. Each video program 1411 receives the joint dictionary from the scene program and stores it in a ring buffer, which maps frame numbers to the returned dictionary. Using the returned key-value dictionary, the video program selects at each moment its subset that approximates the image of the hand associated with the identified subject. These parts of the image frame around the hand joints may be referred to as region proposals.

In the shopping store example, the area proposal is an image frame from the hand position of one or more cameras (with the subject in their respective viewing fields). Zone proposals are generated by each camera in the system. It includes empty-handed and the hand carrying items in the shop's inventory and items that are not in the shop's stock. The video program selects the portion of the image frame that contains the hand joints at each moment. A similar fragment of the foreground mask is generated. The above (image parts of hand joints and foreground masks) are concatenated with joint dictionaries (indicating the subject to which each hand joint belongs) to generate a multi-dimensional array. This output from the video program is provided as input to the WhatCNN model.

The classification results of the WhatCNN model are stored in the region proposal data structure (generated by the video program). All regions for a moment are then provided as input to the scene program. The scene program stores the results in a key-value dictionary, where the key is the subject identifier and the value is a key-value dictionary, where the key is the camera identifier and the value is the logit of the zone. This aggregated data structure is then stored in a ring buffer, which maps the frame number to the aggregated structure at each instant.

WhenCNN with scene and video program

FIG. 20 presents an embodiment of a system in which WhenCNN 1508 receives the output from the scene program, followed by hand image classification performed by the WhatCNN model per video program, as explained in FIG. 19 . A zone proposal data structure for a period of time (eg, for one second) is provided as input to the scene program. In one embodiment, where the camera is capturing images at a rate of 30 frames per second, the input includes 30 time periods and corresponding zone proposals. The scenario program subtracts 30 zone proposals (per lot) for a single integer (which represents an inventory item SKU). The output of the scene program is a key-value dictionary, where the key is the subject identifier and the value is the SKU integer.

The WhenCNN model 1508 performs time series analysis to determine the evolution of this dictionary over time. This results in the identification of items removed from shelves and items placed on shelves in a shopping store. The output of the WhenCNN model is a key-value dictionary, where the key is the subject identifier and the value is the logit generated by WhenCNN. In one embodiment, a set of heuristics 2002 are used to determine the shopping cart data structure 1510 for each subject. Heuristics are applied to the output of WhenCNN, the joint positions of the subject indicated by their respective joint data structures, and the planogram. A planogram is a precomputed map of inventory items on a shelf. Heuristic 2002 determines (for each take or put down) whether the inventory item was placed on or removed from the shelf, whether the inventory item was placed in a shopping cart (or basket) or removed from a shopping cart (or basket) ), or whether the inventory item is in close proximity to the identified subject's body.

Example Architecture of What-CNN Model

FIG. 21 presents an example architecture of WhatCNN model 1506. In this example architecture, there are a total of 26 convolutional layers. The dimensions of the different layers are also presented with respect to the respective width (pixels), height (pixels) and number of channels. The first convolutional layer 2113 receives the input 2111 and has a width of 64 pixels, a height of 64 pixels and has 64 channels (written as 64x64x64). Details of the input to WhatCNN are presented above. The direction of the arrows indicates the flow of data from one layer to subsequent layers. The second convolutional layer 2115 has dimensions of 32x32x64. Following on from the second layer, there are eight convolutional layers (shown in box 2117), each having dimensions of 32x32x64. Only two layers 2119 and 2121 are shown in box 2117 Elucidation of the purpose. This is followed by another eight convolutional layers 2123 with dimensions of 16x16x128. Two such convolutional layers 2125 and 2127 are shown in FIG. 21 . Finally, the last eight convolutional layers 2129 have dimensions of 8x8x256 each. Two convolutional layers 2131 and 2133 are shown in box 2129 for illustration.

There is a fully connected layer 2135 with 256 inputs from the last convolutional layer 2133, which produces N+5 outputs. As noted above, "N" is the number of SKUs that represent "N" unique inventory items available for sale in the shopping store. The five additional logs include: the first logit, which expresses the confidence that the item in the image is a non-SKU item, and the second logit, which expresses the confidence whether the subject holds the item. The next three Logit lines represent the first, second, and third proximity categories, as described above. The last output of WhatCNN is shown at 2137. The example architecture uses batch normalization (BN). The distribution of layers in a Convolutional Neural Network (CNN) changes during training and it varies from layer to layer. This reduces the convergence speed of the optimization algorithm. Batch normalization (Ioffe and Szegedy 2015) is one technique to overcome this problem. ReLU (Corrected Linear Unit) enables the nonlinearity used for each layer except the last output where softmax is used.

22, 23, and 24 are graphical visualizations of different parts of an implementation of WhatCNN 1506. The graphs were adapted from their graph visualization of the WhatCNN model produced by TensorBoard ^™ . TensorBoard ^™ is a suite of visualization tools for viewing and understanding deep learning models (eg, convolutional neural networks).

Figure 22 shows how it detects one hand (the "one hand" model 2210) High-level architecture for convolutional neural network models. The WhatCNN model 1506 contains two such convolutional neural networks to detect the left and right hands, respectively. In the embodiment shown, the architecture includes four blocks, referred to as block 0 2216, block 1 2218, block 2 2220, and block 3 2222. A block is a higher-level abstraction and contains the majority of its nodes representing the convolutional layer. The blocks are arranged in a lower-to-higher sequence such that their output from one block is input to a subsequent block. The architecture also includes an aggregation layer 2214 and a convolutional layer 2212. Between the blocks, different nonlinearities can be used. In the embodiment shown, ReLU nonlinearity is used as described above.

In the embodiment shown, the input to the one-handed model 2210 is the BxWxHxC tensor, which is defined as described in WhatCNN 1506 above. "B" is the batch size, "W" and "H" indicate the width and height of the input image, and "C" is the number of channels. The output of the one-handed model 2210 is combined with the second one-handed model and passed to the fully connected network.

During training, the output of the one-handed model 2210 is compared to the ground truth. The calculated prediction error between the output and the ground truth is used to update the weights of the convolutional layers. In the embodiment shown, Stochastic Gradient Descent (SGD) is used to train WhatCNN 1506.

FIG. 23 presents further details of block 0 2216 of the one-handed convolutional neural network model of FIG. 22 . It contains four convolutional layers, labeled conv0, conv1 2318, conv2 2320, and conv3 2322 in box 2310. Further details of the convolutional layer conv0 are presented in Box 2310. The input is processed by convolutional layer 2312. The output of the convolutional layer is processed by batch normalization layer 2314. ReLU nonlinearity 2316 is applied to batch normalization The output of the ization layer 2314. The output of the convolutional layer conv0 is passed to the next layer conv1 2318. The output of the final convolutional layer conv3 is processed by addition operation 2324. This operation sums the output from layer conv3 2322 to its unmodified input through skip connection 2326. It has been published by He et al. in a paper titled "Recognition Mapping in Deep Residual Networks" (published at https://arxiv.org/pdf/1603.05027.pdf, July 25, 2016) which forwards and The backward signal can be propagated directly from one block to any other block. This signal propagates through the convolutional neural network unchanged. This technique improves the training and testing performance of deep convolutional neural networks.

As shown in Figure 21, the output of the convolutional layers of WhatCNN is processed by fully connected layers. The outputs of the two one-handed models 2210 are combined and passed as input to the fully connected layer. FIG. 24 is an example implementation of a fully connected layer (FC) 2410 . The input to the FC layer is processed by the reshaping operator 2412. The reshape operator changes the shape of the tensor before passing it to the next layer 2420. Reshaping includes flattening the output from the convolutional layer, that is, reshaping the output from a multidimensional matrix into a one-dimensional matrix or vector. The output of the reshape operator 2412 is passed to the matrix multiplication operator, which is designated as MatMul 2422. The output from MatMul 2422 is passed to the matrix positive addition operator, which is designated as xw_plus_b 2424. For each input "x", operator 2424 multiplies the input by a matrix "w" and a vector "b" to produce the output. "w" is the trainable parameter associated with the input "x" and "b" is another trainable parameter called offset or intercept. The output 2426 from the fully connected layer 2410 is a BxL tensor, as explained above in the description of WhatCNN 1506. "B" is the batch size, and "L=N+5" is the number of logit outputs per image frame. "N" is the number of SKUs representing "N" unique inventory items available for sale in the shopping store.

WhatCNN model training

Training datasets of images of hands holding different inventory items in different contexts, and empty hands in different contexts are generated. To accomplish this, human actors hold each unique SKU inventory item in many different ways, at different locations in the test environment. The range of the background of his hands includes: close to the actor's body, close to the store shelf, and close to the actor's shopping cart or basket. The actor also performed the above actions with his bare hands. This procedure is done on both the left and right hands. Most of the actors performed these actions simultaneously in the same test environment to simulate the natural obstructions that would occur in a real shopping store.

The camera 114 captures images of the actors who perform the aforementioned actions. In one embodiment, twenty cameras are used in this process. Joint CNNs 112a-112n and tracking engine 110 process the images to identify joints. The bounding box generator 1504 generates a bounding box similar to the hand region of production or reasoning. Instead of sorting the hand regions via WhatCNN 1506, the images are saved to a storage disk. Saved images are viewed and marked. Images are assigned three labels: Inventory item SKU, background, and whether the hand holds something. This procedure is performed for a large number of images (up to millions of images).

Image files are organized according to data collection scenarios. Naming conventions for image files identify the content and context of those images. Figure 25 shows An image file name in an exemplary embodiment is shown. The first part of the filename (designated by the number 2502) identifies the data collection scene and also includes the time stamp of the image. The second part of the filename 2504 identifies the source camera. In the example shown in Figure 25, the image is captured by "Camera 4". The third part 2506 of the filename identifies the frame number from the source camera. In the example shown, the filename indicates that it is the 94,600th image frame from camera 4. The fourth part 2508 of the filename identifies the extent of the x and y coordinate regions in the source image frame from which the hand region image was obtained. In the example shown, the region is defined between the x-coordinate values from pixels 117-370 and the y-coordinate values from pixels 370-498. The fifth part 2510 of the filename is the personal id that identifies the actor in the scene. In the example shown, the person in the scene has id "3". Finally, the sixth part 2512 of the file name is the SKU number (item=68) that identifies the inventory item, identified in the image.

In the training mode of WhatCNN 1506, the forward pass and the backward pass are performed as opposed to the generation mode, where only the forward pass is performed. During training, WhatCNN generates the categories of the identified subject's hands in the forward pass. The output of WhatCNN is compared to the ground truth. In backward propagation, the gradients of one or more cost functions are computed. The gradients are then propagated to Convolutional Neural Networks (CNN) and Fully Connected (FC) Neural Networks so that their prediction errors are reduced, resulting in outputs that are closer to the ground truth. In one embodiment, Stochastic Gradient Descent (SGD) is used to train WhatCNN 1506.

In one embodiment, 64 images are randomly selected from the training data and augmented. The purpose of image augmentation is to diversify the training data , which leads to better performance of the model. Image augmentation includes random flipping, random rotation, random hue shift, random Gaussian noise, random contrast change, and random cropping of images. The amount of amplification is a hyperparameter and is tuned through a hyperparameter search. Augmented images were classified by WhatCNN 1506 during training. The classification is compared to the ground truth, and the coefficients or weights of WhatCNN 1506 are updated by computing the gradient loss function and multiplying the gradient by the learning rate. The above procedure is repeated multiple times (eg, about 1000 times) to form the epochs. Between 50 and 200 periods are fulfilled. During each epoch, the learning rate is reduced slightly, following a cosine annealing schedule.

WhenCNN model training

The training of WhenCNN 1508 is similar to WhatCNN 1506 described above, using back propagation to reduce prediction errors. Actors perform a variety of actions in a training environment. In an exemplary embodiment, the training is performed in a shopping store with shelves stacked with inventory items. Examples of actions performed by actors include: removing inventory items from shelves, placing inventory items back on shelves, placing inventory items in a shopping cart (or basket), retrieving inventory items from a shopping cart, Swap items between, put inventory items in the actor's hideout. Hideout refers to the location on the body of an actor who can hold inventory items next to his left and right hands. Some examples of hiding places include: an inventory item that is squeezed between the forearm and upper arm, squeezed between the forearm and the chest, squeezed between the neck and shoulders.

Camera 114 records video of all the actions described above during training. The videos are viewed and all frames are marked to indicate Timestamp and action performed. These tags are called action tags for the respective image frames. The image frames are processed through multiple CNN pipelines directly to WhatCNN 1506 as described above for generation or inference. The output of WhatCNN (along with associated action labels) is then used to train WhenCNN 1508 using these action labels as ground truth. Stochastic Gradient Descent (SGD) with cosine annealing schedule was used for the training of WhatCNN 1506 as described above.

In addition to image augmentation (used in the training of WhatCNN), temporal augmentation was also applied to the image frame during the training of WhenCNN. Some examples include mirroring, adding Gaussian noise, transposing logs associated with left and right hands, shortening time, shortening time series by discarding image frames, lengthening time series by duplicating frames, and discarding time series. Data points to simulate imperfections in the base model that produced the input for WhenCNN. Mirroring includes reversing the time series and individual labels, eg, a drop action becomes a take action when reversed.

Use Background Image Processing to Predict Inventory Events

A system and various embodiments for tracking changes by subject in an area of real space are described with reference to Figures 26-28B.

system structure

26 presents a high-level overview of a system in accordance with one embodiment. Since FIG. 26 is an architectural diagram, some details are omitted to improve the clarity of the description.

The system proposed in FIG. 26 receives image frames from multiple cameras 114. As described above, in one embodiment, the cameras 114 may be synchronized with each other in time such that their images are captured simultaneously (or close in time) and at the same image capture rate. The images captured in all cameras which cover the area of real space simultaneously (or temporally close) are synchronized, since their synchronized images can be identified in the processing engine as represented by the fixed in real space Different perspectives of a moment in time on the subject of the location.

In one embodiment, cameras 114 are installed in shopping stores (such as supermarkets) such that a set of cameras (two or more) with overlapping viewing fields are placed over each aisle to capture the real space in the store image. There are "n" cameras in real space. Each camera produces a sequence of images corresponding to the real space of its respective viewing field.

The subject recognition subsystem 2602 (also referred to as the first image processor) processes the image frames received from the camera 114 to identify and track subjects in real space. The first image processor includes a subject image recognition engine. A subject image recognition engine receives a corresponding sequence of images from the plurality of cameras and processes the images to identify subjects represented by images in the corresponding sequence of images. In one embodiment, the system includes a per-camera image recognition engine as described above for identifying and tracking multi-joint subjects. Alternative image recognition engines may be used, including instances in which only one "joint" is identified and tracked per body, or other features covering space and time or other types of image data are utilized to identify and track its processed Subjects in real space.

The "Semantic Difference" subsystem 2604 (also referred to as the second image processor) includes a background image recognition engine that receives a corresponding sequence of images from the plurality of cameras and semantically identifies the background (ie, inventory displays such as shelves). structure) as it relates to, for example, the dropping and removal of inventory items over time in the imagery from each camera. The second image processor receives the output of the subject recognition subsystem 2602 and the image frame from the camera 114 as input. The second image processor shades the identified subjects in the foreground to generate shaded images. The occluded images are generated by replacing their bounding boxes corresponding to the foreground subject with background image data. Continuing here, the background image recognition engines process the masked images to identify and classify background changes represented in the images in the corresponding sequences of images. In one embodiment, the background image recognition engines include convolutional neural networks.

Finally, the second image processor processes the identified background changes for the first set of detections of removal of inventory items by the identified subject and detection of the placement of inventory items on the inventory display structure by the identified subject. The first set of detections is also referred to as the background detection of inventory items put down and removed. In the example of a shopping store, the first detection is to identify an item of inventory that has been removed from or placed on a shelf by a customer or store employee. The semantic difference subsystem includes logic to cause identified context changes to be associated with identified subjects.

The zone proposal subsystem 2606 (also referred to as the third image processor) includes a foreground image recognition engine that receives the corresponding sequence of images from the plurality of cameras 114 and semantically identifies the foreground (ie, shopper, important items in their hands and inventory items) as they relate to, for example, the dropping and removal of inventory items over time in images from various cameras. Subsystem 2606 also receives the output of subject identification subsystem 2602. The third image processors process sequences of images from camera 114 to identify and classify previous changes represented in the images in the corresponding sequences of images. The third image processor processes the identified foreground changes for a second set of detections of removal of inventory items by the identified subject and detection of the placement of inventory items on the inventory display structure by the identified subject. The second set of tests is also referred to as the front-office tests for the drop and take-off of inventory items. In the shopping store example, the second set of detections is to identify the removal of inventory items by consumers and store employees and the placement of inventory items on the inventory display structure.

The system depicted in FIG. 26 includes a selection logic component 2608 for processing the first and second sets of detections to generate a log data structure that includes a list of identified subjects' inventory items. Selection logic 2608 selects the output from either the semantic difference subsystem 2604 or the region proposal subsystem 2606 for take or drop in real space. In one embodiment, selection logic 2608 selects using confidence scores generated by the semantic difference subsystem for the first set of detections and confidence scores generated by the region proposal subsystem for the second set of detections. The output of the subsystem with a higher confidence score for a particular detection is selected and used to generate a log data structure 1510 (also referred to as a shopping cart data structure) that includes a list of inventory items associated with the identified front-office entities.

Subsystem Components

Figure 27 presents the subsystem components that implement the system to track changes by subjects in an area of real space. The system includes a plurality of cameras 114 that generate respective sequences of images of corresponding viewing domains in the real space. The viewing field of each camera overlaps the viewing field of at least one other camera of the plurality of cameras as described above. In one embodiment, a sequence of image frames corresponding to images produced by the plurality of cameras 114 is stored in a circular buffer 1502 (also referred to as a ring buffer) for each camera 114 . Each image frame has a timestamp, an identification of the camera (abbreviated as "camera_id"), and a frame identification (abbreviated as "frame_id"), along with image data. The circular buffer 1502 stores a collection of consecutive time-stamped image frames from the respective cameras 114. In one embodiment, camera 114 is configured to generate a synchronized sequence of images.

The same camera and the same sequence of images are used by both the foreground and background image processors in a preferred embodiment. As a result, redundant detection of put-down and take-out of inventory items is performed using the same input data, allowing high confidence (and high accuracy) in the resulting data.

The subject recognition subsystem 2602 (also referred to as the first image processor) includes a subject image recognition engine that receives corresponding sequences of images from the plurality of cameras 114 . The subject image recognition engine processes images to identify subjects represented in the images in the corresponding sequences of images. In one embodiment, the subject image recognition engine is implemented as a convolutional neural network (CNN), referred to as joint CNNs 112a-112n. The outputs of the joint CNNs 112a-112n corresponding to cameras with overlapping viewing fields are combined to map the positions of the joints from the 2D image coordinates of each camera to the real space. 3D coordinates. The joint data structure 800 of each subject (j), where j equals 1 to x, identifies the positions of the joints of the subject (j) in real space and in the 2D space of each image. Certain details of the master data structure 800 are presented in FIG. 8 .

Background image storage 2704 in semantic disparity subsystem 2604 stores masked images (also known as background images, where foreground subjects have been removed by masking) for the corresponding sequence of images from camera 114 . Background image storage 2704 is also referred to as a background buffer. In one embodiment, the size of the masked image is the same as the size of the image frame in the circular buffer 1502 . In one embodiment, the masked image is stored in the background image store 2704, which corresponds to each image frame in the sequence of image frames per camera.

The semantic disparity subsystem 2604 (or the second image processor) includes a mask generator 2724, which generates a mask of the foreground subject represented by the image in the corresponding sequence of images from the camera. In one embodiment, a mask generator processes the sequence of images per camera. In the shopping store example, the front-office subject is a customer or store employee in front of a background shelf containing items for sale.

In one embodiment, the joint data structure 800 and the image frame from the circular buffer 1502 are provided as inputs to the mask generator 2724. The joint data structure identifies the position of the foreground subject in each image frame. Mask generator 2724 generates a bounding box for each foreground subject identified in the image frame. In this embodiment, the mask generator 2724 uses the x and y coordinates of the joint positions in the 2D image frame to determine the four boundaries of the bounding box. The minimum value of x (all x values from the joints for a subject) is what defines the subject The left vertical border of the bounding box. The minimum value of y (from all y values for a body's joints) defines the bottom vertical boundary of the bounding box. Likewise, the maxima of the x and y coordinates identify the right vertical and top horizontal boundaries of the bounding box. In the second embodiment, the mask generator 2724 uses a convolutional neural network based person detection and localization algorithm to generate the bounding box of the foreground subject. In this embodiment, the mask generator 2724 does not use the joint data structure 800 to generate the bounding box of the foreground body.

Semantic disparity subsystem 2604 (or second image processor) includes masking logic to process images in the sequences of images and replace them with background image data from background images of the corresponding sequences of images to represent the images The subject's foreground image data is identified to provide a masked image, which results in a new background image for processing. When the circular buffer receives an image frame from the camera 114, the masking logic processes the images in the sequences of images to replace the foreground image data defined by the image mask with background image data. The background image data is taken from the background images of the corresponding sequences of images to generate the corresponding masked images.

Consider the example of a shopping store. Initially, at time t =0, when there are no customers in the store, the background image in the background image store 2704 is the same as its corresponding image frame in the sequences of images for each camera. Now considering time t = 1, the consumer moves in front of the shelf to purchase the item in the shelf. Mask generator 2724 generates the bounding box for the consumer and transmits it to mask logic component 2702. Mask logic 2702 replaces pixels inside the bounding box in the image frame at t =1 with corresponding pixels in the background image frame at t =0. This results in an occluded image at t=1 corresponding to the image frame at t = 1 in circular buffer 1502. The occluded image does not include pixels for the foreground subject (or consumer), which are now replaced by pixels from the background image frame at t =0. The occluded image at t =1 is stored in the background image store 2704 and used as the background image for the next image frame at t =2 in the sequences of images from the corresponding cameras.

In one embodiment, masking logic component 2702 combines (such as by averaging or summing by pixels) sets of N masked images in the sequences of images to generate a sequence of factorized images for each camera . In this embodiment, the second image processors identify and classify background changes by processing the sequence of factorized images. The factorized image can be generated, for example, by taking the average of the pixels in the N occluded images in the sequence of occluded images per camera. In one embodiment, the value of N is equal to the frame rate of the camera 114 , eg, if the frame rate is 30 FPS (frames per second), the value of N is 30. In this embodiment, the masked images for a time period of one second are combined to generate a factorized image. Obtaining the average pixel value minimizes pixel fluctuations due to sensor noise and luminance changes in areas of real space.

The second image processors identify and classify background changes by processing the sequence of factorized images. The factorized images in the sequences of factorized images are compared by the bitmask calculator 2710 to previous factorized images of the same camera. Pairs of factorized images 2706 are provided as input to a bitmask calculator 2710 to generate a bitmask that identifies changes in corresponding pixels of the two factorized images. The bitmask has 1 at the pixel position, where between the corresponding pixels (current and previous factorized images) The difference between the RGB (red, green and blue channels) values is greater than the "difference threshold". The value of this difference threshold is adjustable. In one embodiment, the value of the difference threshold is set at 0.1.

The bitmask and the pair of factorized images (current and previous) from the sequence of factorized images per camera are provided as input to the background image recognition engine. In one embodiment, the background image recognition engine includes a convolutional neural network and is referred to as changing CNNs 2714a-2714n. A single-change CNN processes a sequence of factorized images per camera. In another embodiment, masked images from the corresponding sequence of images are not combined. The bitmask is computed from the pairs of masked images. In this embodiment, the pairs of masked images and the bit mask are then provided as input to the change CNN.

The input to changing the CNN model in this example consists of seven (7) channels, including three image channels (red, green, and blue) per factorized image and one channel for the bit mask. The modified CNN includes a plurality of convolutional layers and one or more fully connected (FC) layers. In one embodiment, the modified CNN includes the same number of convolutional and FC layers as the joint CNNs 112a-112n shown in FIG. 5 .

Background image recognition engines (change CNNs 2714a-2714n) identify and classify changes in the factorized images and generate change data structures for the corresponding sequences of images. The change data structures include the coordinates in the masked image of the identified background changes, the identifiers of the inventory item bodies of the identified background changes, and the category of the identified background changes. The categories of identified background changes in the change data structures classify whether the identified inventory item has been added relative to the background image or remove.

Because items can be picked up or put on a shelf by one or more subjects at the same time, the changing CNN produces a number "B" that overlaps the bounding box predictions for each output location. Bounding box predictions correspond to changes in the factorized image. Consider that the shopping store has unique inventory items with the number "C", each identified by a unique SKU. The change CNN predicts the SKU of the body of the inventory item for the change. Finally, the change CNN identifies a change (or inventory event type) for each location (pixel) in the output that indicates whether the identified item was taken from the shelf or dropped on the shelf. The above three pairs of outputs from changing CNN are described by the formula "5*B+C+1". Each bounding box "B" prediction contains five (5) numbers, so "B" is multiplied by five. These five numbers represent the "x" and "y" coordinates of the center of the bounding box, the width and height of the bounding box. The fifth number represents the confidence score of the changed CNN model for the prediction of the bounding box. "B" is a hyperparameter that can be adjusted to improve the performance of changing the CNN model. In one embodiment, the value of "B" is equal to four. Consider that the width and height (in pixels) of the output from the varying CNN are denoted by W and H, respectively. The output of changing the CNN is then expressed as "W*H*(5*B+C+1)". The bounding box output model is based on the object detection system proposed by Redmon and Farhadi in their paper "YOLO9000: Better, Faster, Stronger" (published December 25, 2016). The paper is available at https://arxiv.org/pdf/1612.08242.pdf.

The outputs of the CNNs 2714a-2714n are combined by the coordination logic component 2718 corresponding to changes in the sequence of images from cameras with overlapping viewing domains. The Coordination Logic Component handles data from Change the data structure of groups of cameras to find the identified background changes in real space. Coordination logic component 2718 selects bounding boxes that represent inventory items with the same SKU and the same inventory event type (pick or drop) from most cameras with overlapping viewing fields. The selected bounding box is then triangulated in the 3D real space (using the triangulation techniques described above) to identify the location of the inventory item in the 3D real space. The positions of the shelves in the real space are compared to the triangulated positions of the inventory items in the 3D real space. False positive predictions are discarded. For example, if the triangulated position of the bounding box does not map to the position of the shelf in real space, the output is discarded. The triangulated location of its mapping to the bounding box in the 3D real space of the shelf is considered a true prediction of the inventory event.

In one embodiment, the categories of identified background changes in the change data structures generated by the second image processor classify whether the identified inventory item has been added or removed relative to the background image . In another embodiment, the categories of identified background changes in the change data structures indicate whether the identified inventory item has been added or removed relative to the background image, and the system includes a method for enabling The context changes the logic associated with the identified subject. The system performs detection of the removal of inventory items by the identified entities and detection of the placement of inventory items on the inventory display structure by the identified entities.

The log generator 2720 implements logic to associate a change identified by a true prediction of the change with an identified subject proximate the location of the change. In the embodiment that utilizes the joint recognition engine to identify the subject, the log generator 2720 uses the joint data structure 800 to determine the The position of the main body's hand joints. A subject is identified whose hand joints are within a threshold distance from the changed position at the moment of change. The log generator associates the change with the identified subject.

In one embodiment, as described above, the N masked images are combined to generate a factorized image, which is provided as input to the changing CNN. Consideration: N is equal to the frame rate (frames per second) of the camera 114 . Thus, in this embodiment, the position of the subject's hand during a second time period is compared to the changed position to associate the changes with the identified subject. If more than one subject's hand joint positions fall within the threshold distance from the changed position, the association of the change with the subject is deferred to the output of the foreground image processing subsystem 2606 .

The foreground image processing (region proposal) subsystem 2606 (also referred to as the third image processor) includes a foreground image recognition engine that receives the sequences of images from the plurality of cameras. The third image processors include logic to identify and classify previous changes represented in the images in the corresponding sequences of images. The zone proposal subsystem 2606 generates a second set of detections of taking inventory items by the identified subjects and detections of placing inventory items on the inventory display structure by the identified subjects. As shown in FIG. 27, subsystem 2606 includes bounding box generator 1504, WhatCNN 1506, and WhenCNN 1508. Joint data structure 800 and per-camera image frames from circular buffer 1502 are provided as inputs to bounding box generator 1504 . Details of bounding box generator 1504, WhatCNN 1506, and WhenCNN 1508 were presented earlier.

The system depicted in Figure 27 includes selection logic to process The first and second sets of detections are used to generate a log data structure that includes a list of inventory items for the identified subject. A first set of detections of removal of inventory items by the identified entities and detections of placing inventory items on the inventory display structure by the identified entities is generated by the log generator 2720 . The first set of detections is determined using the output of the second image processor and the joint data structure 800 (described above). A second set of detections of removal of inventory items by the identified subjects and detections of placement of inventory items on the inventory display structure by the identified subjects are determined using the output of the third image processor. For each actual inventory event (taken or put down), the selection logic controller 2608 selects a output. In one embodiment, the selection logic selects the output from the image processor with the higher confidence score for the inventory event.

The program flow of background image semantic difference

Figures 28A and 28B present the detailed steps performed by the semantic disparity subsystem 2604 to track changes in subjects through regions of real space. In the shopping store example, the subjects are the consumers and store employees in the store moving in the aisles between the shelves and other open spaces. The process begins at step 2802. As described above, the camera 114 is calibrated before the sequence of images from the camera is processed to identify the subject. Details of camera calibration are presented above. Cameras 114 with overlapping viewing fields capture images of the real space in which the subject appears. In one embodiment, the cameras are configured to generate a synchronized sequence of images at a rate of N frames per second. of each camera The sequence of images is stored in each camera's respective circular buffer 1502, at step 2804. A circular buffer (also known as a ring buffer) stores a sequence of images in a sliding window of time. Background image storage 2704 is initialized with the initial image frame in the sequence of image frames per camera without foreground subjects (step 2806).

As the bodies move in front of the shelf, a bounding box for each body is generated using its corresponding joint data structure 800 (described above) (step 2808). At step 2810, a masked image is generated by replacing the pixels in the bounding box of each image frame with pixels from the background image store 2704 at the same location from the background image. Masked images for each of the sequences of images corresponding to each camera are stored in background image storage 2704 . The i-th masked image is used as the background image to replace pixels in subsequent (i+1) image frames in the sequence of each camera's image frame.

At step 2812, the N masked images are combined to generate a factorized image. At step 2814, a difference heatmap is generated by comparing the pixel values of pairs of factorized images. In one embodiment, the difference between pixels at positions (x, y) in the 2D space of the two factorized images ( fi 1 and fi 2) is calculated in Equation 1 as follows:

Differences between pixels at the same x and y positions in 2D space are determined using the respective intensity values of the red, green and blue (RGB) channels (as shown in the equation). The above equations provide the value of the difference (also known as the Euclidean modulus value) between corresponding pixels in the two factorized images.

Difference heatmaps can contain noise due to sensor noise and luminance changes in areas of real space. In Figure 28B, at step 2816, a bitmask is generated for the difference heatmap. A semantically meaningful change is identified by a cluster of 1 (one) in the bit mask. These clusters correspond to changes in their identification of inventory items removed from or placed on the shelf. However, noise in the difference heatmap can introduce random 1s into the bitmask. Furthermore, most changes (most items taken from or placed on the shelf) can introduce overlapping clusters of 1. At the next step in the program flow (2818), image shape operations are applied to the bit mask. Image morphological operations remove noise (unwanted 1s) and also attempt to separate overlapping clusters of 1s. This results in a cleaner bit mask containing clusters corresponding to one of the semantically meaningful changes.

Two inputs are provided to the morphological operations. The first input is the bit mask and the second input is called the structural element or kernel. The two basic morphological operations are "erosion" and "dilation". The kernel consists of 1s arranged in a rectangular matrix of various sizes. Kernels of different shapes (eg, circular, oval, or cross-shaped) are generated by adding 0s at specific positions in the matrix. Kernels of different shapes are used for image morphing operations to obtain the desired result when cleaning the bit mask. During the erosion operation, the kernel slides (or moves) over the bit mask. Pixels (either 1 or 0) in the bitmask are considered 1 if all pixels under the kernel are 1s. Otherwise, it is eroded (changed to 0). The erosion operation can be used to remove isolated ones in the bitmask. However, erosion also shrinks the clusters of 1 by eroding the edges.

Dilation is the opposite of erosion. In this operation, when the kernel slides over the bitmask, the values of all pixels in the bitmask area overlapped by the kernel are changed to 1, provided that at least one pixel below the kernel If the value is 1. Dilation is applied to the bitmask after erosion to increase the size clusters by 1. Dilation does not introduce random noise into the bitmask because noise is removed during erosion. A combination of erosion and dilation operations are applied to obtain a cleaner bit mask. For example, subsequent lines of computer code apply a 3x3 filter of 1 to the bitmask to perform an "open" operation, which applies an erosion operation followed by a dilation operation to remove noise and restore the bitmask The size of the cluster of 1, as described above. The above computer code uses the OpenCV (Open Source Computer Vision) library for programming functions of real-time computer vision applications. The library is available at https://opencv.org/._bit_mask=cv2.morphologyEx(bit_mask,cv2.MORPH_OPEN,self.kernel_3x3,dst=_bit_mask)

The "close" operation applies the dilation operation followed by the erosion operation. It can be used to close small holes inside a cluster of 1s. The following code uses a 30x30 cross-shaped filter to apply a close operation to the bitmask.

The bitmask and the two factorized images (before and after) are provided as input to a per-camera convolutional neural network (referred to as a change CNN as above). Changing the output of the CNN changes the data structure. At step 2822, the output from the changing CNN with overlapping viewing domains is used earlier The triangulation techniques described are combined. The changed position in the 3D real space matches the position of the shelf. If the location of the inventory event maps to the location on the shelf, the change is considered a true event (step 2824). Otherwise, the change is a false positive and discarded. The real event is related to the foreground subject. At step 2826, the foreground subject is identified. In one embodiment, the joint data structure 800 is used to determine the position of the hand joints within the threshold distance of the change. If the foreground subject is identified at step 2828, then the change is associated with the identified subject at step 2830. If no foreground subject is identified at step 2828, for example, due to the majority of subjects' hand joint positions within the threshold distance of the change. Then, at step 2832, redundant detection of the change by the zone proposal subsystem is selected. The routine ends at step 2834.

Training Change CNN

A training dataset of seven channel inputs was generated to train the modified CNN. One or more hosts as consumers perform pick-up and drop-off actions by pretending to shop in a shopping store. The main system moves in the aisle, removing inventory items from the racks and placing items back on the racks. Images of actors performing pick and drop actions are collected in circular buffer 1502. The images are processed to generate factorized images, as described above. Pairs of factorized images 2706 and corresponding bit masks output by bitmask calculator 2710 are manually reviewed to visually identify changes between the two factorized images. For a factorized image with a change, a bounding box is manually drawn around the change. This is the smallest bounding box that contains the cluster of 1's corresponding to the change in the bitmask. of the changing inventory item The SKU number is identified and included in the tag for the image (along with the bounding box). An event type identifying the taking or putting down of an inventory item is also included in the label of the bounding box. The label of each bounding box thus identifies (its position on the factorized image) the SKU of the item and the event type. The factorized image can have more than one bounding box. The above procedure is repeated for each change in all the collected factorized images in the training data set. A pair of factorized images (along with the bitmask) form the seven channel inputs to the changing CNN.

During the training of the modified CNN, forward pass and backward pass are performed. In the forward pass, the change CNN identifies and classifies background changes, which are represented in the factorized images in the corresponding sequences of images in the training data set. The change CNN is a first set of processing identified context changes for detection of removal of an inventory item by an identified subject and detection of an inventory item placed by an identified subject on an inventory display structure. During backpropagation, the output of the altered CNN is compared to the ground truth (as indicated in the labels of the training dataset). The gradients of one or more cost functions are computed. The gradients are then propagated to Convolutional Neural Networks (CNN) and Fully Connected (FC) Neural Networks so that their prediction errors are reduced, resulting in outputs that are closer to the ground truth. In one embodiment, a softmax function and a cross-entropy loss function are used for the training of the altered CNN for the class prediction portion of the output. The category forecast portion of the output includes the SKU identifier for the inventory item and the event type (ie, take or put down).

The second loss function is used to train the altered CNN for the prediction of the bounding box. This loss function is calculated between the prediction box and the ground truth box Intersection over union (IOU) between. The region of the intersection of bounding boxes predicted by the changing CNN with the true bounding box label is divided by the region of the union of the same bounding boxes. The value of IOU is high if the overlap between the prediction box and the ground truth box is large. If more than one prediction bounding box overlaps the ground truth bounding box, the one with the highest IOU value is selected to compute the loss function. The details of the loss function were proposed by Redmon et al. in their paper "You Only Look Once: Unified, Real-Time Object Detection" published on May 9, 2016. The paper is available at https://arxiv.org/pdf/1506.02640.pdf.

specific implementation

In various embodiments, the system (as described above) for tracking the placement and removal of inventory items by subject in an area of real space also includes one or more of the following features.

1. District Proposal

The district proposes a framed image of hand positions from all the different cameras it covers the person. Zone proposals are generated by each camera in the system. It includes empty hands as well as hands carrying store items.

1.1 WhatCNN model

Region proposals can be used as input for image classification using deep learning algorithms. This classification engine is called the "WhatCNN" model. It is a classification model in hand. It is the thing in the hands of classification. even part of the object The separation is blocked by the hand, and the image classification in the hand can still be operated. Smaller items can be blocked by hands up to 90%. The area of image analysis by the WhatCNN model is intentionally kept small (in some embodiments) because it is computationally expensive. Each camera can have its own GPU. This is performed for each frame for each hand image from each camera. In addition to the above image analysis by the WhatCNN model, confidence weights are also assigned to the image (one camera, one time point). The sorting algorithm outputs a logit covering the full list of stock keeping units (SKUs) to generate a list of product and service identifiers for the store for n items and an extra for empty hands (n+1).

The scene program now sends its results back to each video program by sending a key-value dictionary to each video. Here, the key is the unique joint ID and the value is the unique personal ID associated with that joint. If no one is associated with the joint, it is not included in the dictionary.

Each video program receives a key-value dictionary from the scene program and stores it in a ring buffer, which maps frame numbers to the returned dictionary.

Using the returned key-value dictionary, the video selects at each moment its subset that approximates the image of the hand associated with the known person. These regions are numpy fragments. We also obtained similar segments around the foreground mask and the original feature array of the articulated CNN. The combined regions are concatenated together into a single multidimensional numpy array and stored in a data structure that holds: the numpy array associated with the region and the person ID, and the region from the individual which hand.

All proposal fields are then fed into the FIFO queue. this queue The coefficients accept the coefficient area and push its numpy array into memory on the GPU.

When the array reaches the GPU, it is fed into a CNN dedicated to classification, called WhatCNN. The output of this CNN is a flat array of floats of size N+1, where N is the number of unique SKUs in the store, and the last category represents no category (or empty hand). The floats in this array are called logs.

WhatCNN results are stored back into the zone data structure.

All regions for a moment are then passed back from each video program to the scene program.

The scene program receives all regions from all videos at a time and stores the results in a key-value dictionary, where the key is the personal ID and the value is a key-value dictionary, where the key is the camera ID And that value is Logit of the District.

This aggregated data structure is then stored in a ring buffer, which maps the frame number to the aggregated structure at each instant.

1.2 WhenCNN model

Images from different cameras processed by the WhatCNN model are combined during a period of time (most cameras during a period of time). An additional input to this model is the hand position in 3D space, triangulated from most cameras. Another input to this algorithm is the distance of the hand from the store's planogram. In some embodiments, a planogram can be used to identify whether the hand is approaching a shelf containing a particular item (eg, cheerios boxes). Another input to this algorithm is the foot part on the store set.

In addition to item sorting using SKUs, a second sorting model uses time series analysis to determine whether the item was picked up from the shelf or placed on the shelf. The images are analyzed over a period of time to determine whether the item, which was in the hand in the previous image frame, has been placed back into the rack or picked up from the rack.

For a second time (30 frames per second) cycle and three cameras, the system will have 90 class outputs, with confidence for the same hand. This combined image analysis significantly increases the chances of correctly identifying the object in the hand. Analysis over time improves the quality of the output, albeit at some very low confidence levels for the individual boxes. This step may have output confidence ranging from, for example, 80% accuracy to 95% accuracy.

The model also includes the output from the shelf model as its input to identify what the person has picked up.

The scenario program waits for 30 or more aggregate structures to accumulate (which represent at least one second of real time), and then performs further analysis to reduce aggregate structures down to a single integer for each person ID-hand pair, where the integer is the Unique ID of the SKU in the store. For a moment, this information is stored in a key-value dictionary, where the key is a personal ID-hand pair and the value is a SKU integer. This dictionary is stored in the ring buffer over time, which maps the number of boxes to each dictionary for that time.

Additional analysis can then be performed to observe how this dictionary changes over time to identify when an individual took something and what he took. This model (WhenCNN) sends SKU Logit and Needle Question to Breen: Something was taken? something was placed? The Logit.

The output of WhenCNN is stored in a ring buffer, which maps the number of boxes to a key-value dictionary, where the key is the personal ID and the value is the extended logit sent by WhenCNN.

Another set of heuristics was then run on both the WhenCNN and the stored results of the person's stored joint positions, as well as on the precomputed map of items on store shelves. This set of heuristics determines where its taking and dropping caused items to be added or removed from. For each take/drop, the heuristics determine that the take or drop is from or to a shelf, from or to a basket, or from or to an individual. The output is the inventory for each person, which is stored as an array, where the array value on the SKU's index is the number of those SKUs that the individual has.

When a shopper approaches the store's exit, the system can transmit an inventory list to the shopper's cell phone. The handset then displays the user's inventory and asks for confirmation to charge from its stored credit card information. If the user accepts, their credit card will be charged. If it does not have a credit card known in the system, it will be asked to provide credit card information.

Alternatively, shoppers may also approach kiosks within the store. The system recognizes when the shopper is approaching the service desk and will send a message to the service desk to display the shopper's inventory. The service desk requires the shopper to accept a charge for the inventory. If the shopper accepts, they can then swipe their credit card or insert cash to pay. Figure 16 presents an illustration of the WhenCNN model for region proposals.

2. Misplaced items

This feature identifies misplaced items when they are individually placed back on a random shelf. This creates problems with object identification, as the foot and hand positions relative to the planogram will be incorrect. Thus, the system builds a modified planogram over time. Based on previous time-series analysis, the system was able to determine whether the individual had put the item back on the shelf. The next time the item is picked up from the shelf position, the system knows that there is at least one misplaced item in the hand position. Accordingly, the algorithm will have some confidence that the individual may pick up the misplaced item from the shelf. If the misplaced item is picked up from the shelf, the system subtracts the item from the location and the shelf no longer has the item. The system can also notify the clerk about the misplaced item via the app so that the clerk can move the item to its correct shelf.

3. Semantic Differences (Shelf Model)

Alternative techniques for background image processing include background subtraction algorithms to identify changes to items on the shelves (items removed or placed). This is based on changes in pixel level. If a person is in front of the shelf, the algorithm stops so that it does not take into account pixel changes due to the presence of a person. Background subtraction is a noise procedure. Therefore, a cross-camera analysis is performed. If enough cameras agree that there is a "semantically meaningful" change in that shelf, the system records that there is a change in that part of the shelf.

The next step is to identify whether the change is "put down" or "Take" changes. For this, time series analysis of the second classification model is used. A zone proposal for that particular portion of the shelf is generated and passed through a deep learning algorithm. This is easier than hand image analysis because the object is not blocked inside the hand. A fourth input is provided to the algorithm, in addition to the three typical RGB inputs. The fourth channel is background information. The output of the shelf or semantic difference is re-input to the second classification model (time series analysis model).

Semantic differences in this approach include the following steps:

1. The image from the camera is compared to an earlier image from the same camera.

2. Each corresponding pixel between the two images is compared via Euclidean distance in RGB space.

3. The distance above some threshold is marked, which results in a new image of the pixel just marked.

4. A set of image shape filters is used to remove noise from the marked image.

5. We then search for a large set of marked pixels and form a bounding box around it.

6. For each bounding box, we then observe the raw pixels in the two images to obtain two image snapshots.

7. The two image snapshots are then pushed into a CNN, which is trained to classify whether the image region represents a removed item or a placed item and what the item is.

3. Store inspection

Inventory of each rack is maintained by the system. It is updated when the item is picked up by the consumer. At any point in time, the system can generate audits of store inventory.

4. Most projects in hand

Different images are used for most projects. Two items in hand are treated differently than one item. Some algorithms can only predict one item rather than a number of items. Thus, the CNN is trained so that it can perform on a "two" number of items different from the single item in hand.

5. Data collection system

Pre-defined shopping scripts are used to collect information on the good quality of the images. These images are used to train the algorithm.

5.1 Shopping Script

Data collection includes the following steps:

1. Scripts are automatically generated to inform human actors what actions to take.

2. These actions are randomly sampled from the set of actions that include: take item X, drop item X, hold item X for Y seconds.

3. When performing these actions, the actor moves and orients itself in as many ways as possible while still being successful at the given action.

4. During the sequence of actions, the set of cameras is recorded from many viewpoints these actors.

5. After the actors have completed the script, the camera video is bundled and stored with the original script.

6. The script acts as an input label to a machine learning model (such as a CNN) trained on the actor's video.

6. Product Line

The system and parts of it can be used for cashierless checkout, which is supported by the following applications.

6.1 Store Apps

Store applications have several major possibilities: providing data analysis visualization, supporting loss prevention, and providing a platform to assist consumers by showing retailers where people are in the store and what items they have collected. Permission levels for employees and access to applications can be determined by the retailer.

6.1.1 Standard Analysis

Data is collected by the platform and can be used in a variety of ways.

1. Derivative data is used to perform various analyses of the store, the shopping experience it offers, and consumer interactions with products, the environment, and others.

a. The data is stored and used in the background to fulfill store and Analysis of consumer interaction. The store application will display some visualizations of this data to the retailer. Other data is stored and queried (when the data point is desired).

2. Heat map:

The platform visualizes the retailer's floor plan, shelf layout, and other store environments, with overlays showing levels of various activities.

1. Example:

1. A map of places where people walk by without touching any products.

2. A map of where people stand when interacting with the product.

3. Misplaced items:

The platform tracks the owner of the store's SKU. When an item is placed in an incorrect location, the platform will know where the item is and build a log. At a certain threshold, or immediately, store employees may be alerted about misplaced items. Alternatively, employees can access misplaced item maps in the store application. Staff can then quickly locate and correct misplaced items when convenient.

6.1.2 Standard assistance

‧The store app will display the floor plan of the store.

‧It will display graphics to represent everyone in the store.

• When the graphic is selected (via touch, press, or other means), relevant information for store employees will be displayed. For example: buy The cart item (the one it has collected) will appear in the list.

‧If the platform has a confidence level below a predetermined threshold for a particular item and for a period of time (as to whether it is owned by someone (shopping cart)), its graph (currently a dot) will indicate the difference. The app uses color changing. Green indicates high confidence and yellow/orange shades indicate lower confidence.

‧Store employees with store apps can be informed about this lower confidence. They can go and confirm that the customer's shopping cart is correct.

‧Through the store app, the retailer's staff will be able to adjust the customer's shopping cart items (add or delete).

6.1.3 Standard LP

‧If the shopper is using the store app, they only need to leave the store and be charged. However, if it is not, it will have to use the guest app to pay for the items in its shopping cart.

• If a shopper bypasses the guest app on his way out of the store, its graphics indicate that he must be approached before leaving. The app uses a color change to red. Personnel also receive notifications of potential damages.

‧Through the store app, the retailer's staff will be able to adjust the customer's shopping cart items (add or delete).

6.2 Non-Store Apps

The following analytical features represent additional capabilities of the platform.

6.2.1 Standard Analysis 1. Product interaction:

Granular breakdown of product interactions, such as:

a. Interaction time and conversion ratio for each product.

b. A/B comparison (color, style, etc.). Some of the smaller products on the display stand have multiple options, such as colors, flavors, and more.

‧Is rose gold manipulated more than silver?

‧Do blue jars attract more interaction than red jars?

2. Directional impression:

Know the difference between location-based impressions and where shoppers are focused. If he is looking at his product at 15 feet away (20 seconds), the impression should not consider where he is, but where he is looking.

3. Consumer identification:

Remember repeat shoppers and their associated email addresses (collected in a variety of ways by retailers) and shopping profiles.

4. Group dynamics:

Determine when shoppers are watching others interact with products.

‧ Did the person interact with the product after answering?

• Did those people enter the store together, or could they be strangers?

‧Individuals or groups of people spend more time in the store?

5. Consumers return to the array:

Provide consumer target information and publish store experience. This feature may have slightly different implementations from retailer to retailer, depending on particular habits and strategies. It may require integration and/or development from retailers to take advantage of this feature.

‧Shoppers will be asked if they would like to receive notifications about products they may be interested in. This step can be integrated with the store's method of collecting emails.

• After leaving the store, the consumer can receive an email with the products they spent time in the store. Interaction thresholds for duration, contact, and sighting (directional impressions) will be determined. When the threshold is met, the products will enter her list and be delivered to her shortly after she leaves the store.

Additionally, or alternatively, shoppers may be sent an email after a period of time offering discounted products or other special information. These products will be items in which they expressed interest (but did not purchase).

6.3 Guest Applications

The shopper app automatically checks people out when they leave the store. However, the platform does not require shoppers to have or use The store must be used by the Inventor app.

When a shopper/person does not have or use the shopper app, they go to a help desk (iPad/tablet or other screen) or they go to a pre-installed self-checkout machine. The display (integrated with the platform) will automatically display the consumer's shopping cart.

Shoppers will have the opportunity to view what it shows. If they agree to the information on the display, they can put cash into the machine (if the capability is built into the hardware (eg, a self-checkout machine)) or they swipe a credit or debit card. They can then leave the store.

If they disagree with the display, store personnel are told to challenge their choice by touch screen, button, or other means. (See Store Assist under Store Apps)

6.4 Shopper App

Through the use of the app (shopper app), the consumer can leave the store with the item and be automatically charged and provided with a digital receipt. Shoppers must open their app at any time while in the shopping area of the store. The platform will recognize its unique image displayed on the shopper's device. The platform will bind them to their account (Consumer Association) and will be able to remember who they are all the time they are in the shopping area of the store, whether or not they keep the app open.

When a shopper collects items, the shopper app will display those items in the shopper's cart. If shoppers want, they Product information can be viewed about each item they have picked up (ie, added to their shopping cart). Product information is stored in the store's system or added to the platform. The ability to update this information, such as offering product discounts or displaying prices, is an option that retailers can request/purchase or develop.

When a shopper drops an item, it is removed from his cart on the backend and on the shopper app.

If the shopper app is opened, and then closed after the consumer association is complete, the platform will maintain the shopper's cart and charge them correctly (once they leave the store).

The shopper app also has mapping information about its development criteria. It can inform the consumer where to find the item in the store if the consumer requests the information by typing in the search item. At a later date, we will take the shopper's shopping list (either manually typed into the app or through some other intelligent system) and display the quickest route through the store to collect all the desired items. Other filters (such as "Bagging Preferences") can be added. The bagging preference filter allows shoppers not to follow the fastest route, but to collect the toughest items first, followed by the more fragile items later.

7. Types of consumers

Member Consumers - The first type of consumers use an app to log into the system. The consumer is prompted with a picture and when she/he presses, the system will link it to the consumer's internal id. If the consumer has an account, the account is automatically charged (when the consumer when leaving the store). This is a member based store.

Guest Consumer - Not every store will have a membership system, or the consumer may not have a smartphone or credit card. Consumers of this type will report to the service desk. The service desk will display the items the customer has and will ask the customer to put money in. The service desk will have been informed about all the items that the consumer has purchased. For this type of consumer, the system can identify if the consumer has not paid for the items in the shopping cart and prompt the cash register at the door (before the consumer gets there) to let the cash register know about the outstanding payment s project. The system may also prompt for an item that has not been paid for, or the system has low confidence about an item. This is called predictive path finding.

The system assigns color codes (green or yellow) to consumers walking through the store based on confidence levels. Green-coded consumers are either logged into the system or have high confidence in the system about them. Consumers coded in yellow have one or more items for which they have not been predicted with high confidence. The clerk can look at the yellow dots and press on them to identify the problem item, walk up to the customer and fix the problem.

8. Analysis

A large group of analytical information is collected about this consumer, such as how much time the consumer spends in front of a particular shelf. In addition, the system tracks where the consumer is looking (impressions about the system), as well as the items that the consumer picks up and puts back on the shelf. Such analytics are currently available for e-commerce but not yet for retail stores.

9. Functional modules

The following is a list of functional modules:

1. Use a systematic capture array of images in a store that synchronizes cameras.

2. A system for identifying joints in images, and sets of joints of individuals.

3. A system for generating new people using joint sets.

4. A system to delete ghost people using joint sets.

5. A system to track individuals over time by tracking sets of joints.

6. A system to generate zone suggestions for each person present in the store, which indicates the SKU number of the item in hand (WhatCNN).

7. A system to perform a get/put analysis for district proposals that indicates whether the item in hand is picked up or put on the shelf (WhenCNN).

8. System to generate per-person inventory arrays using zone proposal and get/put analysis (output of WhenCNN combined with heuristics, person's stored joint positions, and precomputed maps of items on store shelves) .

9. A system for identifying, tracking and updating the location of misplaced items on shelves.

10. A system to track changes (gets/puts) to items on the shelf using pixel-based analysis.

11. A system for performing inventory audits of stores.

12. A system for identifying the majority of items in hand.

13. A system for collecting item image data from stores using shopping scripts.

14. A system to perform checkout and collect payments from member consumers.

15. A system to perform checkouts and collect payments from guest consumers.

16. A system for performing loss prevention by identifying unpaid items in a shopping cart.

17. A system for tracking consumers using color codes to assist store associates in identifying incorrectly identified items in a consumer's shopping cart.

18. A system for generating consumer shopping analytics including location-based impressions, directional impressions, A/B analysis, consumer identification, group dynamics, and the like.

19. A system for generating targeted consumer responses using shopping analytics.

20. A system for generating heatmap overlays of stores to visualize different activities.

The techniques described in this article can support cashierless checkout. go to the shop. take something away. leave.

Cashierless checkout is a pure machine vision and deep learning based system. Shoppers skip the line to get what they want faster and easier. No RFID tags. No changes to the store's backend systems. Can be integrated with ^3rd party Point of Sale and Inventory Management systems.

Real-time 30 FPS analysis of each video feed.

Pre-built, cutting-edge GPU clusters.

Identify shoppers and the items they interact with.

There is no internet dependency in the example embodiment.

Most state-of-the-art deep learning models (including proprietary custom algorithms) to address gaps in machine vision technology for the first time.

Techniques & Capabilities include the following:

1. Standard cognitive machine learning pipeline system to solve:

a) Human detection.

b) Monomer tracking.

c) The majority of cameras personally agree.

d) Hand detection.

e) Item classification.

f) Project ownership resolution.

Combining these techniques, we can:

1. Track everyone throughout their instant shopping experience.

2. Know what shoppers have in their hands, where they stand, and what they put back.

3. Know what direction shoppers are facing and for how long.

4. Identify misplaced items and perform 24/7 visual merchandising audits.

Detects what a shopper actually has in his or her basket.

Learn about your store:

Custom neural networks trained for specific stores and items. Training data can be reused across all store locations.

Standard deployment:

Ceiling cameras must be installed with double coverage of all areas of the store. Between 2 and 6 cameras are required for general walkways.

Pre-built GPU clusters can fit into one or two server racks in the back office.

Example systems may integrate with or include points of sales and inventory management systems.

A first system, method and computer program product for capturing an array of images in a store using synchronized cameras.

A second system, method, and computer program product for identifying joints in an image, and sets of joints of individual people.

A third system, method, and computer program product for generating a new person using joint assembly.

A fourth system, method and computer program product for removing ghost people using joint sets.

A fifth system, method and computer program product for tracking individuals over time by tracking sets of joints.

A sixth system, method and computer program product for generating zone recommendations for each person present in the store, indicating the number of SKUs (WhatCNN) for the item in hand.

To perform the first step of the acquire/drop analysis for the district proposal 7. Systems, methods, and computer program products that indicate whether an item in hand is picked up or placed on a shelf (WhenCNN).

Eighth system, method, and computer program product using area proposal and get/put analysis to generate per-person inventory arrays (eg, with heuristics, person's stored joint positions, and precomputed maps of items on store shelves) combined with the output of WhenCNN).

Ninth system, method and computer program product for identifying, tracking and updating the location of misplaced items on a shelf.

A tenth system, method, and computer program product using pixel-based analysis to track changes (gets/puts) to items on a shelf.

Eleventh system, method and computer program product for performing an inventory audit of a store.

Twelfth system, method and computer program product for identifying a majority of items in hand.

Thirteenth system, method and computer program product for using a shopping script to collect item image data from a store.

Fourteenth system, method and computer program product for performing checkout and collecting payment from member consumers.

Fifteenth system, method, and computer program product for performing checkout and collecting payments from guest consumers.

A sixteenth system, method and computer program product for performing loss prevention by identifying unpaid items in a shopping cart.

Using, for example, color coding to track consumers to assist store associates in identifying incorrectly identified items in a consumer's shopping cart, Series 17 Systems, methods and computer program products.

Eighteenth system, method and computer program product for generating consumer shopping analytics including one or more location-based impressions, directional impressions, A/B analysis, consumer identification, group dynamics, and many more.

A nineteenth system, method and computer program product for generating targeted consumer responses using shopping analytics.

A twentieth system, method and computer program product for generating a heat map overlay of a store to visualize different activities.

Twenty-first system, method and computer program for hand detection.

Twenty-second system, method and computer program for item classification.

Twenty-third system, method and computer program for project title resolution.

Twenty-fourth system, method and computer program for subject person detection.

Twenty-fifth system, method and computer program for item tracking.

The twenty-sixth method and computer program used in the project with the consent of the majority of cameras.

A twenty-seventh system, method and computer program product substantially as herein described for cashierless checkout.

Any of systems 1-26 and any other system or listed above Combination of the systems in systems 1-26.

Described herein is a method for tracking the placement and removal of inventory items by a subject in an area of real space, comprising: using a plurality of cameras to generate respective sequences of images of corresponding viewing areas in the real space, The viewing field of each camera overlaps the viewing field of at least one other camera of the plurality of cameras; receiving the sequences of images from the plurality of cameras and using a first image recognition engine to process the images to generate its identifying a first data set of subjects and positions of the identified subjects in the real space; processing the first data set to specify that it includes a delimitation of images of the identified subjects' hands in the images in the sequence of images boxes; receiving the sequences of images from the plurality of cameras and processing the specified bounding boxes in the images to use a second image recognition engine to generate a classification of the identified subject's hands, the classification Includes whether the identified subject is holding an inventory item, a first proximity category indicating the position of the identified subject's hand relative to the shelf, and a second proximity category indicating the identified subject's hand relative to the identified subject's hand. the position of the identified subject's body, a third proximity class indicating the position of the identified subject's hand relative to the basket associated with the identified subject, and an identifier for possible inventory items; and the processing of the identified subject's image The hand class of a collection of images in these sequences to detect the taking of inventory items by the identified subject and the placing of inventory items by the identified subject on the inventory display structure.

In the methods described herein, the first data sets may be for Each identified subject contains a set of candidate joints with coordinates in real space.

The method of this description can include processing the first sets of data to specify bounding boxes including specifying bounding boxes based on positions of joints in the sets of candidate joints for each subject.

In the methods described herein, one or both of the first and second image recognition engines may include convolutional neural networks.

The methods of this description may include using a convolutional neural network to process the classes of bounding boxes.

Describes a computer program product (and product) comprising computer readable memory comprising a non-transitory data storage medium in which computer instructions stored can be executed by a computer to track an area of real space by Items of inventory of the entity are put down and removed by any of the procedures described herein.

A system is described that includes a plurality of cameras that generate sequences of images including a subject's hands; and a processing system coupled to the plurality of cameras, the processing system including a hand image recognition engine that receives the sequences of images Actions by the subject are identified by generating the class of the hand in the time series, and logic to process the classes of the hand from the sequences of images, where the action is the placing and taking of inventory items one.

The system may include logic to identify the positions of the joints of the subject in the images of the sequences of images, and to identify corresponding images that include the hands of the subject based on the identified joints middle the bounding box.

The Computer Program Listing Appendix is attached to this specification and includes portions of examples of computer programs used to implement portions of the systems provided in this application. This appendix includes examples of heuristics for identifying body joints and inventory items. This appendix presents computer code for updating a subject's shopping cart data structure. This appendix also includes computer programming routines for calculating the learning rate during the training of convolutional neural networks. This appendix includes computer program routines to store the results of the subject's hand classification from the convolutional neural network in a per-subject per-hand data structure per frame from each camera.

112a-112n: Image recognition engine

114: Camera

800: Data Structure

1502: Circular buffer

1504: Bounding Box Generator

1506: WhatCNN

1508:WhenCNN

1510: Shopping Cart Data Structure

2602: First Image Processor Subsystem

2604: Second Image Processor Subsystem

2606: Third Image Processor Subsystem

2608: Select Logic Components

2702: Mask Logic Components

2704: Background image storage

2706: Factorized Imagery

2710: Bit Mask Calculator

2714a-2714n: Changes to CNN

2718: Coordination Logic Components

2720: log generator

2724: Mask Generator

Claims (30)

A system for tracking a multi-joint subject in an area of real space, comprising: a plurality of cameras, the cameras of the plurality of cameras produce respective sequences of images of corresponding viewing fields in the real space, the viewing a field overlapping the viewing field of at least one other camera of the plurality of cameras; and a processing system coupled to the plurality of cameras, the processing system comprising: an image recognition engine receiving images from the plurality of cameras The sequences, which process the images to generate corresponding arrays of joint data structures, the arrays of joint data structures corresponding to a particular image by the joint type, the time of the particular image, and the coordinates of the elements in the particular image to classify the elements of the particular images; a tracking engine configured to receive the arrays of joint data structures corresponding to images in a sequence of images from cameras having overlapping viewing fields, and to correspond to the arrays of joint data structures in a different sequence transform the coordinates of the elements in the arrays of the joint data structure of the image into candidate joints with coordinates in the real space; and logic to convert the set of candidate joints with coordinates in the real space Recognized as a multi-joint body in this real space. The system of claim 1, wherein the image recognition engines comprise convolutional neural networks. The system of claim 1, wherein the image recognition engines process images to generate a confidence array for elements of the images, wherein the confidence array for a particular element of the image includes confidence values for a plurality of joint types of the particular element , and selects the joint type of the joint data structure for the particular element according to the confidence array. The system of claim 1, wherein the logic for identifying the set of candidate joints includes a heuristic function for identifying the set of candidate joints as multiple based on physical relationships between joints of subjects in the real space Joint body. The system of claim 4 includes logic for storing the sets of joints identified as the multi-joint bodies, and wherein the logic for identifying the set of candidate joints includes logic for determining Whether a candidate joint identified in an image taken at a particular time corresponds to a member of one of the sets of candidate joints that were identified as candidate joints of a multi-joint subject in a previous image. The system of claim 1, further comprising logic to process the sets of candidate joints identified as the multi-joint bodies over time to detect the identified multi-joint bodies with real Interaction events for inventory items in an area of space. The system of claim 1, wherein the plurality of cameras comprises cameras disposed above and having viewing fields covering respective portions of the region in real space, which are identified as candidate joints of a multi-joint body The coordinate systems in the real space of the members of the set identify locations in the region of the real space of the articulated body. The system of claim 1 includes logic for tracking the position of a plurality of articulated bodies in the region of real space. The system of claim 8 includes logic for determining when a multi-joint body of the plurality of multi-joint bodies leaves the region of the real space. The system of claim 1 , including logic to track the location in the region of real space of a majority of candidate joints that are members of a set of candidate joints identified as a particular multi-joint body. A method for tracking a multi-joint subject in an area of real space, comprising: using a plurality of cameras to generate respective sequences of images of corresponding viewing fields in the real space, the viewing fields of each camera being associated with the plurality of cameras the viewing fields of at least one of the other cameras overlap; images in the sequences of images are processed to generate respective arrays of joint data structures, the arrays of joint data structures corresponding to particular images are classify the elements of the particular images by joint type, the time of the particular image, and the coordinates of the elements in the particular image; assign the elements in the arrays of joint data structures corresponding to images in different sequences transforming the coordinates of the elements into candidate joints having coordinates in the real space; and identifying the set of candidate joints having the coordinates in the real space as a multi-joint body in the real space. The method of claim 11, wherein the processing the image comprises using a convolutional neural network. The method of claim 11, wherein the processing the images includes generating a confidence array for elements of the images, wherein the confidence array for a particular element of the image includes confidence values for a plurality of joint types of the particular element, and according to the confidence Array to select the joint type for this joint data structure for this particular component. The method of claim 11, wherein the identifying the set of candidate joints comprises applying a heuristic function according to the physical relationship between the joints of the subject in the real space to identify the set of candidate joints as a multi-joint subject. The method of claim 14, comprising storing the sets of joints identified as multi-joint bodies, and the set of candidate joints identified therein The combination includes determining whether a candidate joint identified in an image taken at a particular time corresponds to a member of one of the sets of candidate joints that were identified as candidate joints of a multi-joint subject in a previous image. The method of claim 11, wherein the sequences of the image are synchronized. The method of claim 11, wherein the plurality of cameras comprise cameras disposed above and having viewing fields covering respective portions of the region in real space, which are identified as candidate joints of a multi-joint body The coordinate systems in the real space of the members of the set identify locations in the region of the real space of the articulated body. The method of claim 11 of the claimed scope includes tracking the position of the plurality of multi-joint bodies in the region of the real space. The method of claim 18, comprising determining when a multi-joint body of the plurality of multi-joint bodies leaves the region of the real space. The method of claim 11 , comprising tracking the location in the region of real space of a majority of candidate joints that are members of a set of candidate joints identified as a particular multi-joint body. A computer program product comprising: Computer-readable memory, which includes a non-transitory data storage medium; and computer instructions stored in the memory, which can be executed by the computer to track, by a program, a multi-joint body in an area of real space, the The procedure includes: using a sequence of images from a plurality of cameras having corresponding viewing fields in the real space, the viewing field of each camera overlapping the viewing field of at least one other camera of the plurality of cameras; processing the image images in the sequences to generate corresponding arrays of joint data structures corresponding to a particular image by the joint type, the time of the particular image, and the coordinates of the elements in the particular image the elements of the particular images; transform the coordinates of the elements in the arrays of joint data structures corresponding to images in different sequences into candidate joints having coordinates in the real space; and will have the A set of candidate joints for coordinates in real space is identified as a multi-joint body in that real space. The product of claim 21, wherein the processing the image includes using a convolutional neural network. The product of claim 21, wherein the processing the images includes generating a confidence array for elements of the images, wherein the confidence array for a specific element of the image includes confidence values for a plurality of joint types of the specific element, and according to the confidence array to select the joint information for the particular element The joint type of the material structure. The product of claim 21, wherein the identifying the set of candidate joints comprises applying a heuristic function according to the physical relationship between the joints of the body in the real space to identify the set of candidate joints as a multi-joint body. The product of claim 24, comprising storing the sets of joints identified as multi-joint bodies, and wherein identifying the set of candidate joints includes determining whether the candidate joints identified in images obtained at a particular time conform to It is identified as a member of one of the sets of candidate joints of the multi-joint subject in the previous image. The product of claim 21, wherein the sequences of the image are synchronized. The product of claim 21, wherein the plurality of cameras include cameras disposed above and having viewing fields covering respective portions of the region in real space, which are identified as candidate joints of a multi-joint body The coordinate systems in the real space of the members of the set identify locations in the region of the real space of the articulated body. The product of claim 21 of the scope of application includes tracking the position of a plurality of multi-joint bodies in the area in real space. The product of claim 28 of the claimed scope includes determining when a multi-joint body of the plurality of multi-joint bodies leaves the region of the real space. The product of claim 21, comprising tracking the position in the region of real space of a majority of candidate joints that are members of a set of candidate joints identified as a particular multi-joint body.

Images