WO2023026277A1

WO2023026277A1 - Context-based moniitoring of hand actions

Info

Publication number: WO2023026277A1
Application number: PCT/IL2022/050905
Authority: WO
Inventors: Omer SHMUELI; Danny Albocher; Izhar RAVID
Original assignee: Pickey Solution Ltd.
Priority date: 2021-08-22
Filing date: 2022-08-18
Publication date: 2023-03-02

Abstract

A method, apparatus and computer program product for monitoring hand actions, by a wearable device configured to be worn on a hand of a subject. A first visual input and a second visual input are obtained from a sensor located on the wearable device in a location and orientation enabling monitoring of hand activity of the subject. A determination that the subject is holding an object in the first visual input, may is performed based on an identification of the object in a foreground of the first visual input. A hand-based action of the subject with respect to the object may be automatically identified based on a difference between the first visual input and the second visual input.

Description

CONTEXT-BASED MONITORING OF HAND ACTIONS

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of U.S. provisional Application No. 63/235,805, entitled "CONTEXT-BASED MONITORING OF HAND ACTIONS", filed August 22, 2021, which is hereby incorporated by reference in its entirety, without giving rise to disavowment.

TECHNICAL FIELD

[0002] The present disclosure relates to wearable devices in general, and to wearable devices for monitoring hand actions, in particular.

BACKGROUND

[0003] Actions of workers within the logistic chain, such as stocking, picking or packing in logistic centers or groceries suffer from inaccuracies and inefficiencies. The average error rate related to human actions is 0.5-3%. Such statistic means that at least one mistake will be found at one of any 200 orders. As a result, there is a continuous tension between errors reduction and productivity - faster performance causes higher error rate and vise verse: in order to reduce the error rate, work slows and productivity decreases accordingly.

[0004] New technologies are developed to support better productivity by increasing efficiency (reduce time per task) and accuracy (reduce errors volume below the common rate).

BRIEF SUMMARY

[0005] One exemplary embodiment of the disclosed subject matter is a method comprising: obtaining a first visual input and a second visual input from a sensor located on a wearable device, the wearable device is worn by a subject on a hand of the subject, the sensor is configured to be placed in a location and orientation enabling monitoring of hand activity of the subject; determining that the subject is holding an object in the first visual input. Said determining that the subject is holding the object is based on an identification of the object in a foreground of the first visual input; and automatically identifying a hand-based action of the subject with respect to the object based on a difference between the first visual input and the second visual input.

[0006] Optionally, said automatically identifying the hand-based action of the subject with respect to the object comprises: identifying that the object is not in a foreground of the second visual input.

[0007] Optionally, the method comprises determining that the subject is holding the object in the second visual input based on an identification of the object in a foreground of the second visual input.

[0008] Optionally, said automatically identifying the hand-based action of the subject with respect to the object comprises: determining a first context of the object based on a background of the first visual input; determining a second context of the object based on a background of the second visual input; and identifying the hand-based action of the subject with respect to the object based on a difference between the first context and the second context.

[0009] Optionally, said automatically identifying the hand-based action of the subject with respect to the object further comprises: determining that a physical location of the object has changed based on the difference between the first context and the second context; and determining the hand-based action based on the change of the physical location of the object.

[0010] Optionally, said determining the second context is further performed based on a third input obtained from a second sensor located on the wearable device.

[0011] Optionally, the background of the first visual input is visually similar to the background of the second visual. Optionally, the difference between the first context and the second context is a non-visual difference. [0012] Optionally, the non-visual difference between the first context and the second context is determined based on a difference between a first value associated with the first visual input and a second value associated with the second visual input. Optionally, the first and the second values are determined based on a third input obtained from a second sensor.

[0013] Optionally, the first visual input comprises at least a first portion of the object; wherein the second visual input comprises at least a second portion of the object. Optionally, said automatically identifying the hand-based action of the subject with respect to the object comprises: identifying that the second portion of the object blocks a background of the second visual input.

[0014] Optionally, the second visual input is obtained prior to the first visual input. The second visual input comprises the object; determining that the object appears in a background of the second visual input; and wherein the hand-based action of the subject with respect to the object is a pick-up action of the object by the subject.

[0015] Optionally, the second visual input succeeds the first visual input, wherein the second visual input comprises the object. Optionally, the method further comprises determining that the object appears in a background of the second visual input; and wherein the hand-based action of the subject with respect to the object is a placement action of the object by the subject.

[0016] Optionally, said determining that the object appears in the foreground of the first visual input is performed based a sequence of visual inputs that comprise the first visual input, wherein said determining that the object appears in the foreground comprises: identifying a pattern of appearance of the object in the sequence of visual inputs.

[0017] Optionally, said determining that the object appears in the foreground of the first visual input is performed based a sequence of visual inputs that comprise the first visual input, wherein said determining that the object appears in the foreground comprises: identifying a movement pattern of the hand of the subject with respect to the object in the sequence of visual inputs.

[0018] Optionally, the subject is a picker tasked with picking items to fulfill an order of a customer.

[0019] Optionally, the method further comprises: identifying that the object is associated with the order of the customer; wherein said automatically identifying the hand-based action of the subject with respect to the object comprises: identifying that the picker picked up the object and placed the object in a tote associated with the order of the customer; and in response to said automatically identifying: performing a fulfillment- related action.

[0020] Optionally, the method further comprises identifying a mismatch between the object and a list of items in the order of the customer; wherein said automatically identifying the hand-based action of the subject with respect to the object comprises: identifying that the picker picked up the object and placed the object in a tote associated with the order of the customer; and in response to said automatically identifying: issuing an alert to the picker indicating the mismatch.

[0021] Optionally, a view of the sensor is blocked, at least in part, by the hand.

[0022] Another exemplary embodiment of the disclosed subject matter is a method comprising: obtaining visual inputs from a sensor located on a wearable device, the wearable device is worn by a subject on a hand of the subject, the sensor is configured to be placed in a location and orientation enabling monitoring of hand activity of the subject, a view of the sensor is blocked, at least in part, by the hand, the visual inputs comprise a first visual input, a second visual input, a third visual input and a fourth visual input, the first visual input precedes the second visual input, the third visual input precedes the fourth visual input; automatically identifying a pick action by the hand with respect to an object based on the first visual input and the second visual input, wherein said automatically identifying the pick action is based on a determination that in the first visual input the hand not touching the object and based on a determination that in the second visual input the hand touches the object; and automatically identifying a release action by the hand with respect to the object based on the third visual input and the fourth visual input, wherein said automatically identifying the release action is based on a determination that in the third visual input the hand touches the object and based on a determination that in the fourth visual input the hand does not the object.

[0023] Optionally, the method further comprises based on said automatically identifying the pick action and based on said automatically identifying the release action, updating a digital mapping of objects to locations, wherein said updating updates a location of a digital representation of the object from a first location to a second location. [0024] Optionally, the method further comprises determining a first context of the object based on a background of the first visual input or the second visual input, the first context is associated with the first location; and determining a second context of the object based on a background of the third visual input or the fourth visual input, the second context is associated with the second location.

[0025] Optionally, the second location is determined based on the second context.

[0026] Optionally, the first visual input precedes the second visual input by no more than a first predetermined threshold, the third visual input precedes the fourth visual input by no more than a second predetermined threshold.

[0027] Optionally, the first predetermined threshold and the second predetermined threshold are identical.

[0028] Optionally, the first predetermined threshold or the second predetermined threshold are selected from a group consisting of: no more than 1 second; no more than 500 milliseconds; no more than 200 milliseconds; no more than 100 milliseconds; no more than 50 milliseconds; no more than 30 frames; no more than 10 frames; no more than 5 frames; and no more than a single frame.

[0029] Yet another exemplary embodiment of the disclosed subject matter is a computerized apparatus having a processor, the processor being adapted to perform the steps of: obtaining a first visual input and a second visual input from a sensor located on a wearable device, the wearable device is worn by a subject on a hand of the subject, the sensor is configured to be placed in a location and orientation enabling monitoring of hand activity of the subject; determining that the subject is holding an object in the first visual input, wherein said determining that the subject is holding the object is based on an identification of the object in a foreground of the first visual input; and automatically identifying a hand-based action of the subject with respect to the object based on a difference between the first visual input and the second visual input.

[0030] Y et another exemplary embodiment of the disclosed subject matter is a computer program product comprising a non-transitory computer readable storage medium retaining program instructions, which program instructions when read by a processor, cause the processor to perform a method comprising: obtaining a first visual input and a second visual input from a sensor located on a wearable device, the wearable device is worn by a subject on a hand of the subject, the sensor is configured to be placed in a location and orientation enabling monitoring of hand activity of the subject; determining that the subject is holding an object in the first visual input, wherein said determining that the subject is holding the object is based on an identification of the object in a foreground of the first visual input; and automatically identifying a hand-based action of the subject with respect to the object based on a difference between the first visual input and the second visual input.

THE BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

[0031] The present disclosed subject matter will be understood and appreciated more fully from the following detailed description taken in conjunction with the drawings in which corresponding or like numerals or characters indicate corresponding or like components. Unless indicated otherwise, the drawings provide exemplary embodiments or aspects of the disclosure and do not limit the scope of the disclosure. In the drawings:

[0032] Figure 1 shows a schematic illustration of a monitoring wearable device, in accordance with some exemplary embodiments of the disclosed subject matter;

[0033] Figure 2 shows a schematic illustration of an exemplary self-service shopping or online- shopping fulfillment for customers scheme, in accordance with some exemplary embodiments of the disclosed subject matter;

[0034] Figure 3 shows schematic illustrations of visual inputs provided by a hand action monitoring wearable device, in accordance with some exemplary embodiments of the disclosed subject matter;

[0035] Figures 4A-4B show flowchart diagrams of methods, in accordance with some exemplary embodiments of the disclosed subject matter; and

[0036] Figure 5 shows a block diagram of a system, in accordance with some exemplary embodiments of the disclosed subject matter.

DETAILED DESCRIPTION

[0037] One technical problem dealt with by the disclosed subject matter is to enable monitoring actions performed by a human subject without affecting performance of such actions. The actions may be natural actions, incidental actions, intentional actions, or the like. The actions may be performed by the hand of the human subject, using instruments, tools, or the like. The actions may be performed on objects, on other subjects, or the like. Monitoring hand actions may be essential in various disciplines, starting from enforcing standard operational procedures (SOPs) or safety inspection such as in factories, hospitals, logistic centers, groceries, or the like, to automating actions such as in picking, stocking or self-service shopping. As an example, in self-service shopping, a shopper can enter a store (such as a supermarket, grocery store, fashion store, warehouse, logistics center, or the like), collect items using her hands and perform other related actions. As such, the shopper may purchase products without being checked out by a cashier, without requiring additional scanning of the products before existing the store, or the like. As another example, a picker in warehouse, can collect items using her hands and perform other related actions. As such, the picker’s actions, including inaccurate or inefficient actions, may automatically be captured and analyzed. As yet another example, a worker, such as a picker, or the like, can use or collect items using her hands and perform other related actions. As such, the worker’s actions, may automatically be digitally footprinted, documented or the like, without requiring special actions, gestures, or the like.

[0038] In some exemplary embodiments, different actions of human subjects may be required to be continuously monitored, in order to maintain safety requirements, to supervise the actions, inspect the actions, or the like. As an example, the action of providing drugs to patients performed by health-care staff, may be desired to be monitored and supervised, in order to verify that a correct dose was provided for a patient, to register the timing of providing the drug, to coordinate between the actions performed by different staff members, such as different nurses in different shifts, observing side effects, or the like. As another example, the action of assembling components to a machine performed by a worker, may be desired to be monitored and supervised, in order to verify that the worker follows the machine manual, other operational procedures, or the like.

[0039] In some exemplary embodiments, actions may be monitored using sensors configured to observe the human subject, her action, gestures, related objects, or the like. The observing sensors may be located within the environment of the human subject or the location the actions are supposed to be performed. In addition to requiring special equipment and high additional costs, such monitoring may be limited due to limited ability of capturing the actions from different angles, limited ability of tracking the human subject in different locations, limited ability of tracking multiple subjects simultaneously, limited ability in crowded stores or in situations where customers block the view of other customers, or the like. Additionally or alternatively, monitoring the actions using sensors observing the human subject may affect the privacy of the human subject and other subjects in her environment. Specifically, monitoring from afar the human subject can also capture and monitor the subject's private encounters, conversations, or the like. Referring to the above-mentioned example, monitoring health-care staff may not only violate the privacy of the health-care staff members, but also patients treated thereby.

[0040] Yet another technical problem dealt with the disclosed subject matter is to enable fulfillment of home shopping by human pickers. In some exemplary embodiments, the picker may be tasked with picking items selected by the customer and preparing them for shipment. The picker may pick the items in the store itself, in a storage, in a warehouse, or the like. The picker may be required to read digital or printed orders, including a list of items, pick up the items, place the items in the correct tote associated with each order, transfer the totes for delivery, or the like. Current models of home shopping may involve disadvantages for both the retailers and the customers. On the one hand, the customer may not be able to dynamically update her shopping list, to inspect products before being shipped to her, wrong items may be included in the order and require the customer's review of the supplied items as well as to contact, or the like. On the other hand, the retailer cannot provide an updated items list at each time point for the customer, wrong items may be picked and resulting in reducing cost-effectiveness for the retailer, or the like. Additionally or alternatively, pickers may not be able to efficiently fulfill more than one order simultaneously, due to potential mix-ups and confusions.

[0041] One technical solution is to automatically track and identify hand-based actions of the subject with respect to objects based on visual inputs obtained from sensors of wearable smart device worn by the human subject. The visual input may be analyzed to identify the object in a foreground of the visual input and determine identify hand-based actions of the subject with respect to objects based on differences in contexts and background in different visual inputs (e.g., images). [0042] In some exemplary embodiments, the wearable smart device may be worn on the hand of the human subject, on the wrist of the human subject, or the like. The wearable smart device may be equipped with a vision sensor, such as a digital camera, a radar sensor, a radio waves-based sensor, a laser scanning sensor, a LiDAR, an Infrared (IR) sensor, an ultrasonic transducer, or the like. The vision sensors may be configured to be placed in a location and orientation enabling monitoring of hand activity of the subject, such as by observing the interior portion of the hand of the human subject, other portions of the hand of the human subject, an area surrounding the hand of the human subject, a 360-degree view of the hand, or the like. Based on visual input provided by the wearable smart device, the object grasped by the hand, and context thereof may be identified. The action performed on the object may be identified, and a responsive action may be performed accordingly.

[0043] In some exemplary embodiments, the wearable smart device may be a device that can be worn on human's hand without substantially affecting actions performed by the hand, such as a bracelet, a wristband, a watch, a glove that covers all or part of the hand (e.g., a few fingers, a finger cover), any hand wearable, or the like. Additionally or alternatively, the wearable smart device may be embedded in a smart watch or other wearable device of the human subject being observed. The wearable smart device may be worn in a single hand, in both hands separately, in both hands simultaneously, or the like.

[0044] It is noted that in some embodiments, a body- worn device that is not worn on the hand may be utilized. As an example, the device may be worn on a collar, on a torso, or the like, while having a view of the hand. Such embodiments may be less reliable than hand-worn devices, and may be susceptible to manipulations by the subject. In case of pickers and other staff members, such devices may be utilized when considering that the subject is considered reliable.

[0045] In some exemplary embodiments, the wearable smart device may be configured to identify when the hand of the human subject touches an item, picks the item up, moves the item from one location to another, releases the item, places the item, pressing on the item, unwrapping the item, or the like. Additionally or alternatively, the wearable smart device may be configured to provide a digital footprint of such actions. The device may be further configured to identify the item being touched by the hand(s), such as the type of the item, shape, name, or the like. Additionally or alternatively, the wearable smart device may be configured to identify other attributes related to the item, such as weight, size, color, temperature, texture, expiration date, or the like. In some exemplary embodiments, the wearable smart device may be configured to identify when the action that the hand(s) perform on an item (e.g., touches, holds, grabs, releases, or the like) based on the visual input provided by the vision sensor. The wearable smart device may be configured to identify the item using the visual input. As an example, the device may be configured to identify the item based on an optical image, based on QR code, barcode, any combination of letters, numbers or images, chip, RFID, or the like. In some exemplary embodiments, computer vision techniques may be employed to analyze the images. The image analysis may be performed on-device. Additionally or alternatively, an off-device analysis may be implemented, such as to preserve battery and reduce computation requirements from the device, or the like.

[0046] In some exemplary embodiments, the wearable smart device or system utilizing thereof may be configured to identify in real-time, an origin from which the object is picked, such as a shelf, a box, a container, a storge location, or the like, or a target destination in which the object is being placed in during the stocking, assembly, picking or the shopping session, such as a shelf, a bin, a bag, a cart, a picking pallet, a box, a container, the hands of the subject, or the like. The visual input of the wearable smart device may be analyzed to identify a surface from which the object is picked or on which the object is being placed, a predetermined shopping cart such as a physical shopping cart of the store, a personal shopping bag, or the like. In some exemplary embodiments, the wearable smart device may be configured to identify a candidate shopping cart or a picking pallet, and validate such candidate to be the shopping cart or the picking pallet during the shopping or picking session, such as based on additional visual input, such as later visual input, or the like. As an example, in response to identifying a container in which a first object is placed on, the container may be determined to be a candidate shopping cart. In response to identifying that other objects are being placed in the container by the subject, the container may be validated as the shopping cart. As another example, the container may be determined to be a component in the shopping carts, such as a bag placed inside the shopping carts, a section in a divided shopping cart, or the like. As yet another example, the target destination may be the customer’s hands, which may be considered a container. As yet another example, the container may be determined as a temporary shopping cart upon moving objects placed therein to a different shopping cart. In some exemplary embodiments, the identified shopping cart may be container that is unique and was never before used as a shopping cart. For example, the customer’s personal bag may be a unique bag that was never encountered before. The shopping cart may be identified by first identifying it as a candidate due to it being a container. In view of repetitive insertion of items to the candidate, in view of the candidate being in the view of the sensor during the shopping session over time, or in view of other considerations, the shopping cart candidate may be validated. It may be noted that the shopping or picking cart may dynamically change during the shopping or picking session, such as because of placing objects in different containers during the shopping or picking session, keeping the objects in the hands for a while before being placed in the shopping or picking cart, or the like. A virtual shopping or picking cart may be updated to be associated with each determined shopping or picking cart utilized during the shopping or picking session. An additional validation may be performed to determine that the virtual cart comprises a correct combination of items purchased by the subject and placed in different containers.

[0047] In some exemplary embodiments, some potential containers may be illegal or unauthorized to be used as a rule or for specific items. For example, a pocket in the pants of the user may be a prohibited container and if the user places an item in the pocket, an alert may be triggered. Additionally or alternatively, the pocket may be considered as a prohibited container for some items and allowed for others. As an example, the user may place a pen and her wallet in her pocket, which may be considered a permitted operation. However, if the user places a gemstone in her pocket, an alert may be triggered. Additionally or alternatively, the user may place item A in a container identified by its color, size or shape and may not place item B in this container following manuals, orders or instructions determined for item A, item B and that container.

[0048] In some exemplary embodiments, the one or more sensors may be functionally coupled to a learning module, such as implemented using Artificial Neural Networks (ANN), supervised learning, or the like, configured to analyze photos captured by the sensors and infer information therefrom. Additionally or alternatively, the wearable smart device may comprise a chip or another hardware technology configured to receive, collet and process pictures, videos signals, or the like as captured from the sensors of the wearable smart device. Additionally or alternatively, the retail smart wristband may comprise a transmitter utilized for transmitting input captured by the sensors of the wearable smart device to a backend solution. Such transmitter may use Wi-Fi, Bluetooth, cellular data transmission or the like for transmitting the data. In some exemplary embodiments, a local memory may be utilized to retain information prior to transmission, such as to allow transmission when connectivity is available, to allow re-transmission of the same information in different resolution levels (e.g., initial transmission of low- resolution video to preserve power and bandwidth resources, and re-transmission of high- resolution video of segments of interest in the data). In some exemplary embodiments, transmission may be performed in real-time. Additionally or alternatively, the transmission may be performed in near-real time, such as within a predetermined latency period (e.g., 1 minute, 5 minutes, 10 minutes, or the like). Additionally or alternatively, transmission may be performed in specific timings or upon occurrences of events, such as the user wanting to exit a predetermined perimeter, performing a checkout operation, or the like.

[0049] In some exemplary embodiments, the visual inputs received may be analyzed to distinguish between the item and a context of the item. In some exemplary embodiments, the item may be identified as appearing in the foreground, while the context may be determined based on the background. In some exemplary embodiments, continuous monitoring of an item being moved from one location to another may be performed. In some exemplary embodiments, moving the hand towards an item or opposite of an item may be an indication that this item as the foreground and the surrounding of this item as the background. In some exemplary embodiments, the item at the foreground may remain the same, however, the background may change and it may be determined that the item was moved. In some exemplary embodiments, a first image may be analyzed to determine an item at a first context. A second image of the same item may be analyzed to determine the item once again and a second context. Based on a change in context, the disclosed subject matter may be enabled to determine that an action was performed by the user on the item. Such analysis may overcome technical challenges, such as a difficulty in identifying the item in all images, difficulty in identifying the action being performed, or the like. In some exemplary embodiments, as long as the item is being held by the user, the images captured by the visual sensor would include the item in a relatively constant location, posture and size. When the item is placed, it seizes from being at the foreground of the image, and its visual characteristics, such as size, posture, location in the frame, or the like, may be modified. Based on such a change, an automated identification of an action may be determined. In some exemplary embodiments, as long as the item is being held by the user, the images captured by the visual sensor would include the item in a relatively constant location and the user’s hand which covers part of the item.

[0050] In some exemplary embodiments, once a context of an item is changed, it may be determined that the physical location of the item was changed. For example, when a user picks up an item from a shelf, the initial image may show the item in the background. Once the item moves to the foreground (e.g., picked up by the hand), the context of the item may change to the "hand". Background information from the image may show that the item is being held by the user. Once the user places the item in a tote, the item may appear in the foreground of the image obtained from the sensor, and the background information of the image may indicate that the context is the tote of the user. In some exemplary embodiments, only after the item is placed in the tote and released, the action may be determined, such as based on a change in the images in which the item no longer appears in the foreground.

[0051] It is noted that the identification of an action may be performed based on two images in which the item is identified at different contexts. The change in context may indicate the identified action. For example, changing from shelf to hand may be identified as "picked up", changing from hand to shelf may be identified as "placed in shelf", changing from hand to tote may be "placed in tote", changing from shelf to tote (e.g., without first identifying "picked up" may be identified as "placed in tote", and the like.

[0052] In some exemplary embodiments, the change of context of an item may be captured from the hand perspective. As an example, during observation of the target item and the context thereof, a movement of the hand towards an item may be identified when the item is captured as static object from the hand’s perspective, it may be indicative that the item is picked by the hand. The sensors from the hand’s perspective may continue to observe the item and its context, while being carried by the hand. Accordingly, the item, or a portion thereof, may be observed as static or constant at the hand while the background, if not hidden by the item, may be changing. Similarly, the context of the item may be determined based on capture of the environment surrounding the item. An indication that the hand is moving from that item, placing the item in another location, or the like, may be determined based on capturing a different, equivalent or same background from the hand’s perspective.

[0053] In some exemplary embodiments, the subject may be required to wear two wearable smart devices, one on each hand, in order to monitor actions of the two hands of the subjects. In some exemplary embodiments, a single wearable device that comprises two components that can be disconnected may be utilized. The single wearable device may be worn by the subject on one hand, or can be divided into the two components each of which may be worn on a different hand. Each component may be associated with a sensor that may be configured to continuously monitor the respective hand and provide visual input thereof.

[0054] In some exemplary embodiments, the images and other sensor data captured by the two wearable devices may be combined to provide a more complete understanding of the surroundings. As an example, a same item may be captured by the sensor readings of both devices. If the two devices were independent, in such a case two items may be identified. However, as the two wearable devices are utilized to analyze a same environment, it may be identified that the same item was captured by both, and a single item and a context related thereto may be identified. Additionally or alternatively, in case readings from each device are processed individually, duplicate items may be identified automatically, e.g., by a server processing the information or by a local processor, and one of which may be removed or, as an example supported by action time stamp, movement sensors, vision or the like.

[0055] Another technical solution is utilizing the wearable smart device for retail uses. The wearable smart device may be worn by a customer, a store picker, or the like, to identify items being collected by the customer or the store picker. In some exemplary embodiments, the wearable smart device may be configured to identify when the customer's or picker’s hand touches an item, picks the item up, moves the item, releases the item, places the item on a shopping bag or a shopping cart, places the item back to a location in the store, or the like. The device may be configured to identify a shopping cart in which purchased items are being placed therein, such as based on analysis of the customer's environment, analysis of the customer's behavior, analysis of movement patterns of the customer, or the like. The device may be further configured to identify the item being touched by the customers’ hand(s), such as the type of the item, shape, name, or the like. Additionally or alternatively, the device may be configured to identify other attributes related to the item, such as weight, number of units in the package, size, expiry date, or the like. The device may be personal, e.g., a property of the customer that may be adapted to fit to any store that the customer enters, or a property of the store, that may be provided to each customer that enters the store and personalized or paired to the customer accordingly.

[0056] Yet another technical solution is utilizing the wearable smart device for determining geospatial location of objects in changing environments. In response to determining based on the visual input of the wearable device that the object was picked from one location and placed in another location, a mapping of items in a facility may be updated to indicate the object is located at the other location. As an example, the wearable device may be utilized by the store staff, such as by the retailer, the workers, or the like, in order to create a three-dimensional mapping of the products in the store. The device may be worn by the workers when arranging the products in the store. The three- dimensional mapping may be utilized for multiple uses, such as for identifying the items based on their location, for store uses such as stock check, or the like. Additional information may be fed to the device by the worker during the arrangement, such as the barcode, information about the item, new items, or the like. Additionally or alternatively, input may be provided via a secondary device, such as a terminal connected to the device, a mobile app coupled to the device, or the like.

[0057] Yet another technical solution is utilizing the wearable smart device for manual fulfillment of a shopping order of a customer by a picker. In some exemplary embodiments, the shopping order may comprise a list of items selected by the customer, such as in an on-line shopping platform, in a website of the retailer, or the like. A picker that picks items for the customer may wear the smart wearable device. The visual input obtained from sensors located on the wearable device to identify when the picker picks up an object and places it in a tote associated with the shopping order of the customer. A corresponding item matching the object in the list of items may automatically be marked as fulfilled. Additionally or alternatively, in response to identifying a mismatch between the object and the list of items, such as that the object is not comprised by the list of items, the picker may be alerted of the mismatch. Additionally or alternatively, the picker may perform multiple picking tasks for multiple customers simultaneously. The picker may utilize multiple wearable devices, each of which associated with a different customer and paired to a respective tote within the cart. Additionally or alternatively, the picker may utilize a single wearable device. In some exemplary embodiments, it may be identified into which tote the item was placed, and accordingly update the corresponding customer order, so that the same picker may pick items for multiple customers simultaneously. It may be noted that the wearable device for the manual fulfillment of a shopping order of a customer by a picker, may be configured to be worn on other organs of the picker, such as on the chest, collar, forehead or otherwise mounted on the head of the picker, or the like, in a manner enabling capturing actions of the hands of the picker. Such embodiments may provide a wider scene capturing the hand actions and the objects, being more comfortable for the picker when fulfilling multiple orders simultaneously, or the like. In such embodiments, concerns such as violating privacy of the picker, preventing identification of the customer, or the like, may not be influential.

[0058] In some exemplary embodiments, when a picker starts a fulfilment session, the picker may put the wearable smart device on. The wearable smart device may be automatically or manually synchronized to a designated computing device of the picker, to correlate with customer order being collected. The wearable smart device may be configured to identify any item that the picker picks up and puts into the shopping bag or cart, keeps it within his hand, moves to the other hand, put in a different shopping bag, or the like. A digital shopping list may be automatically created, updated and checked without the need for stopping at checkout point, or being re- reviewed an item by an item, or the like.

[0059] Additionally or alternatively, the picker may utilize a designated a container for each customer order, herein referred to as a shopping cart, that connects to the wearable smart device, or is otherwise associated therewith. The shopping cart may comprise a wireless communication module enabling direct communication with the device (e.g., via Bluetooth), indirect communication therewith (e.g., via a Wi-Fi, via the Internet, or the like), or the like. Additionally or alternatively, the shopping cart may comprise a screen that interfaces to the wearable smart device and displays the items being added to the cart, the total cost of the shopped items, or any other information related to the items, communicate with the customer, or the like. The shopping cart interface may be utilized to verify that the item that the customer is being charged is entered into the correct bill, e.g., to prevent a situation that the customer is being charged for an item not inserted into the shopping cart (as an example, when the picker is fulfilling another order, picks an item from the floor, picks an item from a different cart, enters the item to a wrong cart (e.g., a cart of another customer), or the like). Additionally or alternatively, a mobile application may be utilized to provide a similar display to the customer, to enable communication between the customer and the picker, or the like. [0060] In some exemplary embodiments, the wearable smart device may be configured to identify when the picker 's hand(s) perform an action on an item (e.g., touches, holds, grabs, releases, or the like) based on touch contact between the device and the item, the device and other items, or the like. As an example, the device may comprise pressure sensors that may be located on certain locations thereof that may be in contact with the items, such as three fingers (e.g., on the thumb, index finger and middle finger), or the like. The device may identify, using the pressure sensors that the customer's hand holds an item, releases the item, or the like. For example, existence of pressure may be indicative of the customer holding the item, while lack of pressure may indicate that the item is released. Additionally or alternatively, the device may comprise accelerometer sensors, configured to identify and monitor the movement of the hand. The device may be configured to identify that the item is being held, moves, inserted into the cart, or the like, based on the movement pattern of the picker 's hand. It may be noted that the pressure sensors or the accelerometer sensors may be utilized for determining additional attributes of the item, such as weighing using the pressure sensors, size and number using the accelerometer sensors, or the like. Additional sensors may be utilized, such as, temperature sensors, scanners, or the like

[0061] In some exemplary embodiments, based on the visual input, the item and the action being performed thereon may be identified. The device may comprise a vision sensor such as a camera, an IR sensor, a radar, a LiDAR, an ultrasonic transductor, electro-magnetic waves-based sensor, a laser-based sensor, a visible-light based sensor, an invisible light-based sensor, a combination thereof, or the like, that provides visual representation of the item. It is noted that the visual input may be a visual representation as would be captured by a human being, data that is computationally processed to generate an image (e.g., by an imaging process), or the like. The device may be configured to identify the item using the visual data. As an example, the device may be configured to identify the item based on an optical image, based on QR code, barcode, an identifier of the item, such as a combination of numbers, letters, icons, or the like. In some exemplary embodiments, computer vision techniques may be employed to analyze the images, such as object recognition techniques, image analysis techniques, machine learning, or the like. In some exemplary embodiments, the image analysis may be performed on-device. Additionally or alternatively, off-device analysis may be performed to preserve battery and reduce computation requirements from the device. [0062] In some exemplary embodiments, the device may comprise location sensors that may be configured to identify the item and the action based on the location of the item. The exact location of the item may be determined by a triangulation of known locations inside the store, using a three-dimensional mapping of the store shelves, or the like. In some exemplary embodiments, location sensors may be Radio Frequency (RF)-based, may be based on GPS, may be based on cellular towers, indoor-beacons, or the like. In some exemplary embodiments, the location sensors may be adapted for indoor usage, such as may be based on triangulation of signals provided within the store by designated hardware. Additionally or alternatively, tagging techniques may be utilized to identify an item. As an example, a Radio Frequency Identification (RFID) sensor may be utilized to read RFID information from a tag embedded in or coupled to the item. The RFID information may comprise an indication of the identity of the tagged item, such as a barcode, a sequence of letters, numbers, icons, or the like. As another example, a location of the item may be identified based on a recognition of shelves inside the store, such as based on an identifier of a shelf the item is located thereon, an identifier of an adjacent shelf, recognition of the area the shelf is located therein, or the like.

[0063] In some exemplary embodiments, the device may be configured to apply computerized learning to improve the identification of items based on features that can be learned from sensors, such as shape, weight, temperature, or the like, or the type of the action, such as based on the movement pattern, acceleration, or the like. The information may be verified using the scanner or camera (such as inside the stores) to accurately identify the item.

[0064] In some exemplary embodiments, commencement or performance of scanning or receiving a visual input may be performed by natural nonintentional action of the user or by a gesture or intentional action thereof, such as movement of one or more fingers, which is captured by the camera or alike or through identification of certain data such as capturing any barcode or QR code.

[0065] It may be noted that any action that is attributed to the customer, may also be performed by a picker or any other worker, shopper, or user, picking the items instead of, in behalf or for the customer; and thus, wearing the wearable device.

[0066] Yet another technical solution is utilizing the wearable smart device for validating that the manual fulfillment of a picking order by a picker is performed in accordance with predetermined instructions. In some exemplary embodiments, the picker may be required to put items on a certain location or order at the tote or shopping cart, such as avoiding placing some types of items above or below other items, placing heavy items below light weighted items, placing refrigerated items in a temperature keeping tote, building a stable pile of items within the tote, or the like. Additionally or alternatively, the picker may be required to build a stable pile of items within the tote or the package being delivered to the customer, such as by placing items of certain shape in a certain order, placing larger items below smaller items, supporting items, or the like. The wearable smart device may be configured to validate such building.

[0067] Yet another technical solution is utilizing the wearable smart device for documenting or tracking the manual fulfillment of a picking task by a picker for the purpose of restoring the order of placing items in the tote or to document their placement. In some exemplary embodiments, such documentation may be utilized by the picker or the customer to find items located in the tote or restore the place where certain items were located.

[0068] One technical effect of utilizing the disclosed subject matter is enabling an efficient hand action tracking without violating privacy of monitored subjects and surrounding subjects. The disclosed subject matter may enable monitoring mainly the interior portion of the hand, while a wider scene may be blocked by the hand. Personal identification features, such as in the face, name tags, or the like, may not be captured by the utilized sensors. Furthermore, the data obtained by sensors utilized in the disclosed subject matter may be limited and focused only on information essential for determining the action and the object the action being performed on. The disclosed subject matter may spare tracking and monitoring the entire environment (such as the entire store, the entire hospital room, the entire lab, or the like), and thus reducing costs, not requiring changes in the design or additional equipment, not revealing sensitive data in the environment, or the like. In some exemplary embodiments, the disclosed subject matter may enable receiving visual insights from the hand position, which may be referred to as “hands vision”. The hands vision concept can be used by employers to monitor their workers actions.

[0069] Another technical effect of utilizing the disclosed subject matter is providing for a reliable self-checkout shopping service, with an enhanced user experience for customers in retail stores. Utilizing the disclosed subject matter enables the customer to perform a fast and efficient self-service shopping, while reducing the time that the consumer spends at the store by avoiding waiting in line, avoid scanning of items, reducing billing time, or the like. Furthermore, the disclosed subject matter provides economic benefits to the customers, as the disclosed subject matter enables maintaining competitiveness in the market, reduce labor costs and increase profitability of retailers which may lead to lowering prices, or the like. Additionally or alternatively, the disclosed subject matter provides a seamless shopping experience, lacking a feeling of actual payment. The disclosed subject matter provides such benefits while preserving the privacy of consumers, with no intrusive tracking of the consumer, without capturing face or personal images, or the like.

[0070] Yet another technical effect of utilizing the disclosed subject matter is providing for a healthier self-shopping experience. Utilizing the disclosed subject matter enables avoiding the health risk associated with waiting in lines for check-out points, minimizing physical interaction with store workers, only the customer touches the items while shopping, the customer uses her shopping bag, no need to physically pay using cashier machines or passing money or card to other people, or the like. By providing such benefits, the disclosed subject matter contributes to prevention of spreading infectious diseases, lower viral contagion, and the like.

[0071] Yet another technical effect of utilizing the disclosed subject matter is providing for a reliable self-checkout shopping service, with an affordable price to the retailer side. By enabling an improved consumer shopping experience, the disclosed subject matter may improve consumer attraction and satisfaction, leading to more consumers hiding the retailer store. By reducing the purchase time for each consumer, additional consumers can be served. Furthermore, the disclosed subject matter enables redeploying staff personnel to enhance direct customer service, maximum floor space, saving labor costs, or the like.

[0072] Yet another technical effect of utilizing the disclosed subject matter for the retailer side, is enabling the service in crowded stores without limiting the number of shoppers that can simultaneously shop in the store. While regular cashier-free stores, such as stores providing self-service shopping based on in-store monitoring equipment, may operate with a limited number of customers inside the store, such as because one customer may block the view of other customers, block the view of the monitoring equipment, or the like; the disclosed subject matter may enable self-service shopping also in crowded store, such as stores with a maximal number of customers that can fit in. [0073] Yet another technical effect of utilizing the disclosed subject matter is providing for a manner of collecting data by the consumers, that may be utilized for advertising and enhancing business plans, without bothering the consumers or harming their privacy. The data collected by the disclosed subject matter may be utilized to improve store operations and checkout experience, to merge consumers online and offline identities, to re-target shoppers online based on in-store purchases, to learn shopper interests and habits, to learn consumers’ reaction to various shelf displays and store layouts, or the like. Additionally or alternatively, the data collected by the disclosed subject matter may be utilized to extract data useful for retailer operation. As an example TRAX™ is a system the employs computer vision technology, artificial intelligence, fine-grained image recognition, and machine learning engines to convert store images into shelf insights. As an example, TRAX™ is able to recognize products that are similar or identical, such as branded drinks or shampoo bottles but can also differentiate between them based on variety and size. One effect of the disclosed subject matter may be to collect visible data that can be used to be analyzed by TRAX™ or other similar products, without the need to send dedicated personal or sensors. Instead, the data is collected in a crowd -sourcing methodology and as a side-effect to the customers’ and pickers’ regular activities. It may be noted that in some cases, customers' analytics services may also be provided based on data gathered while the customers use the device, such as shopping session times, shopping habits, preferred routes, alternative goods that the customer considered, identifying lack of confidence in the selection of a good, or the like.

[0074] Yet another technical effect of utilizing the disclosed subject matter is automatically generating an accurate store- wide mapping of items and their locations. In some cases, the mapping may be generated automatically and iteratively according to actions of clerks, workers, pickers, packers, shoppers, or the like, and reflect in real-time and accurately the location of each item in the store.

[0075] Yet another technical effect of utilizing the disclosed subject matter is to automatically provide for a safety system that monitors human activity and identifies violations of safety rules. Such system may achieve a reduction in accidents and incidents caused due to human error. In some exemplary embodiments, the wearable devices may be utilized for safety regulations for workers from different disciplines. Some workers may be obliged to perform certain actions in certain order, while keeping safety regulation procedures. The disclosed subject matter can monitor the hands actions and verify that all required actions are being actually made and in the predefined order. As an example, an aircraft mechanic may be obliged to perform dozens of actions at the aircraft. Safety regulations require that the entire set of actions will be made and in certain order. Other than that mechanic’s reports, his supervisor cannot assure that all these actions are fully made and in the right order. By monitoring hands actions, through analysis of the visual output taken from the hand position, the supervisor can assure compliance of these actions with predefined safety regulations. In some exemplary embodiments, the analysis may be performed automatically and only where there are potential violations the supervisor may manually inspect the video. Additionally, or alternatively, the recording may be inspected per the supervisor's discretion. As an example, upon investigating a safety event, such as a crash or mid-air malfunction, the supervisor may retroactively inspect the video to review whether there was a human-error involved from the point of view of the mechanic.

[0076] Yet another technical effect of utilizing the disclosed subject matter is reducing and preventing thefts, e.g., by workers. In some exemplary embodiments, workers may handle precious items, objects or components which can be easily hidden within their cloths or on their body. The hands vision solution can monitor any action made for the purpose of hiding items, objects or components. An Artificial Intelligence (Al)-based solution may recognize set of actions which are predefined as suspected actions (or restricted actions). As an example, a worker in a cannabis site can easily hide in his gloves, pockets or on his body a portion of the crops. When the crop is unique intellectual property, then one seed or a small stem may include the entire IP developed. As another example, a worker in gemstone mine, factory or lab, can easily conceal precious stones within his cloths or on his body. The disclosed subject matter may be utilized to prevent such activity without requiring constant manual oversight of the workers. It is noted that the disclosed subject matter may be implemented with respect to any precious item, such as but not limited to gemstones, diamonds, gold, platinum, uranium, cannabis, drugs, or the like. A "precious item" in accordance with the disclosed subject matter is an item having high-market value for small portions, such as values over 50 USD (per USD value at 2020) for a volume of less than 3 cubic centimeter.

[0077] Yet another technical effect of utilizing the disclosed subject matter is increasing workers efficiency. The disclosed subject matter may be utilized to monitor actual actions and order of actions, including the time it takes to perform these actions. In some exemplary embodiments, it may be possible to compare between workers’ actions and performances. If few workers do the same job, there should be the most efficient way to do it. The disclosed subject matter may learn these actions, and conclude the most efficient way to perform them. It may also be useful to analyze individual performance of each worker, of sets of workers, and provide an analytics service for worker's activity.

[0078] In some exemplary embodiments, the wearable devices in accordance with the disclosed subject matter, may track workers actions in many places. As example, in cannabis sites, it can be within the green house, in storage rooms, or in the lab. While moving between separated places within the site, the wearable devices may transmit data over local Wi-Fi, cellular network, or other network connectivity means or use local storage until connectivity is restored. Moving between these separated places within the site may be conditioned by wearing wristbands (e.g., otherwise doors will not open). In addition, in certain places, such as restrooms, entrance will shut down the camera and exit will renew its operation. In some exemplary embodiments, before shutting down the camera for privacy issues, the worker must perform an action, such as deposit everything she holds (e.g., a bag with cannabis). As another example, the image capture may continue to operate in private areas but may be marked private and may be only automatically analyzed. In some exemplary embodiments, only in case of a breach or an event that is being investigated, such images may be manually inspected.

[0079] In some exemplary embodiments, switching the wearable device from on to off mode, and vice versa, may depend on pre-configuration. The pre-configuration may be provided by the entity controlling the device, such as the third party, the mining company, the store, or the like. In some exemplary embodiments, the wearable device may be used as a worker tag. As an example, a condition for entering a certain place (door open) is that the worker is wearing 2 wristbands and that they are both functioning properly. A condition for leaving a certain place will be that the wristband has not recognized any suspected action or was functioning properly the entire period of staying at that place.

[0080] In some exemplary embodiments, when a wearable device is not functioning properly, the worker and the employer will receive a real time warning or feedback. For example, the employer will receive it within the employer's dashboard and the worker will be notified through a red light flashing on the wristband. Examples of the wristband not functioning properly may include, for example, camera's lens is covered with dust or otherwise its view is blocked, the camera malfunctioning, the wristband tamper detection is activated, lack of connectivity, storage full, or the like.

[0081] The disclosed subject matter may provide for one or more technical improvements over any pre-existing technique and any technique that has previously become routine or conventional in the art. Additional technical problems, solutions and effects may be apparent to a person of ordinary skill in the art in view of the present disclosure.

[0082] Referring now to Figure 1 showing a schematic illustration of a hand action monitoring wearable device, in accordance with some exemplary embodiments of the disclosed subject matter.

[0083] In some exemplary embodiments, Wearable Device 120 may be a wearable digital product that may be worn by a subject on a Hand 130 of the subject. Wearable Device 120 may be worn on a wrist of Hand 130, such as a smart watch, smart wristband, or the like. Wearable Device 120 may be adapted in size and shape to fit to a human hand or wrist, such as Hand 130. Wearable Device 120 may be worn on the left hand, on the right hand, on a single hand, on both hands (e.g., comprising two wristbands, one on each hand), or the like. Additionally or alternatively, Wearable Device 120 may be embedded in another wearable device of the user, such as a smart watch, a bracelet, or the like.

[0084] In some exemplary embodiments, Wearable Device 120 may comprise one or more vision Sensors 110 located thereon. Sensor 110 may be configured to be placed in a location and orientation enabling monitoring of activity of Hand 130. Sensor 110 may be used to stream live video, to record step-by-step instructions for performing a task, or the like. Sensor 110 may be positioned at a location enabling capturing a view of the inner portion of Hand 130, the palm of the hand, the base portion of the fingers, or the like, such that when a subject is holding an object, Sensors 110 may capture the object being held, at least partially. In some exemplary embodiments, Sensor 110 may be located at the base of the palm of Hand 130, such as at the wrist, or the like. In some exemplary embodiments, Sensor 110 may be a Point Of View (POV) camera designed to capture the scene in front of Hand 130, such as a stationary mounted camera, or the like. Additionally or alternatively, Sensor 110 may be a SnorriCam camera adapted to fit Hand 130, to face Hand 130 directly so that Hand 130 appears in a fixed position in the center of the frame. The SnorriCam camera may be configured to present a dynamic, disorienting point of view from perspective of Hand 130. [0085] Additionally or alternatively, Sensor 110 may comprise several sensors (not shown) embedded in Wearable Device 120, attachable thereto, or the like. As an example, the several sensors may be located all over Wearable Device 120, that cover a full range of view around the hand, such as 360°. As another example, the several sensors may be dispersed non-uniformly over Wearable Device 120, in order to provide the full range of view, provide a view enabling identification of actions and items, or the like. As an example, the several sensors may be located in the portion of Wearable Device 120 that is configured to face the interior portion of Hand 130. The several sensors may be in a predetermined constant distance from each other, may overlap, or the like.

[0086] In some exemplary embodiments, Sensor 110 may comprise visual sensors such multiple camera lenses, different cameras, LiDAR scanners, ultrasonic transductors, RF- based sensors, other sensors or components having alternative or equivalent technology, a combination thereof, or the like. Sensor 110 may be configured to capture pictures, videos or signals around Wearable Device 120. Other types of input may be provided, such as heat maps, thermal images, or the like.

[0087] Additionally or alternatively, Wearable Device 120 may comprise motion sensors or detectors configured to recognize any movement of Wearable Device 120 and support tracking disposition of an item, such as a GPS sensor, an accelerometer, or the like.

[0088] In some exemplary embodiments, Wearable Device 120 may be utilized to recognize that Hand 130is about to perform an action (or is performing the action) on an item, to identify the item being held by Hand 130, information thereabout, or the like. Wearable Device 120 may be utilized to track actions of Hand 130, items Hand 130 performs or avoids performing the action thereon, or the like. Sensor 110 may be configured to recognize when the hand is approaching an object, picking, holding (e.g., the object stays constant at the hand), moving the object (e.g., background picture changed), releasing the object, or the like. Additionally or alternatively, Sensor 110 may be configured to identify parameters of the item or enable identification thereof, such as type, category, name, shape, size, price, or the like. In some exemplary embodiments, Wearable Device 120 may be configured to identify a hand-based action that is not intended as a command to the device itself (e.g., a gesture intended as a purposeful command). As an example, Wearable Device 120 may be utilized to identify a picking up action performed naturally, as opposed to a purposeful gesture with Hand 130 that may be performed specifically with the intent to instruct the device. In some exemplary embodiments, Wearable Device 120 may be configured to identify actions that are performed as part of the regular interaction of the subject with the items, and no dedicated actions or gestures by the subject are relied upon. In some exemplary embodiments, Wearable Device 120 may be configured to identify actions that are performed as a result of a gesture captured by the device, such as gesture, movement or special position of the device user’s finger(s).

[0089] In some exemplary embodiments, Sensor 110 may comprise one or more motion sensors or detectors. Input from the motion sensors may be utilized to support tracking disposition of items upon which the hands perform actions. The motion sensors may be configured to recognize any movement of Wearable Device 120.

[0090] Additionally or alternatively, Sensor 110 may comprise a barcode scanner. Barcode scanner may be utilized to scan barcodes associated with items to support identification thereof, provide additional information, such as price, weight, or the like. Sensor 110 may comprise a barcode scanner together with other sensors, such as a visual sensor (camera) or instead of other sensors.

[0091] Additionally or alternatively, Wearable Device 120 may comprise a Communication Component 140, such as a chip or another hardware technology, configured to receive, collet and process pictures, videos signals, or the like captured by Sensor 110. Additionally or alternatively, Communication Component 140 may comprise a transmitter utilized for transmitting input captured by Sensor 110 to a backend device configured to perform the respected analysis. Such transmitter may be configured to utilize a wireless connection, such as Wi-Fi network, Bluetooth, RF transmission, IR transmission, cellular data transmission, or the like, for transmitting the data. It may be noted that all functionalities of Wearable Device 120 may be based on on-device computations or on off-device computations, such as performed by an edge device, a remote server, a cloud-based server, or the like.

[0092] In some exemplary embodiments, Wearable Device 120 may comprise an output means, such as an Input/Output (I/O) Component 150. I/O Component 150 may be connected to Communication Component 140, Analysis Component 160, or other components of Wearable Device 120 or an operating system thereof. RO Component 150 may be utilized to obtain input or provide output from the subject or other user, such as for informing that the item is identified, viewing a list of items provided by one or more customers, viewing picking tasks status, or the like. I/O Component 150 may comprise a small screen, a microphone, a Light Emitting Diode (LED), or the like. As an example, a green light may be lightened as positive signal. Other types of signals, such as audio signals, vibration signals, or the like, may be used. In some exemplary embodiments, I/O Component 150 may be configured to provide output to the subject (e.g., LED lighting up in green) indicating of an update of her virtual cart, such as in view of an addition of an item thereto. Additionally or alternatively, I/O Component 150 may be configured to provide output to the subject (e.g., LED lighting up in red) indicating of an invalidating of her virtual cart, such as in view of an misidentification of an item in the shopping cart, identification of a tampering event, placing an item in a wrong shopping cart, or the like. Additionally or alternatively, I/O Component 150 may be a more sophisticated touch screen, that may be utilized to provide output and obtain input to and from other users, similar to a smart phone screen, a smart watch screen, or the like.

[0093] In some exemplary embodiments, in case I/O Component 150 is indicative of an alert (e.g., red LED light) or in case I/O Component 150 is not indicative of successful operation (e.g., green LED light), the user may be prevented from performing some actions. As an example, a barrier may prevent the user from exiting the perimeter until all issues are resolved, a door may remain locked prevent exiting from the store until Wearable Device 120 indicates no remaining issues, or the like. In some exemplary embodiments, the determination may be based on communication with Wearable Device 120. Additionally or alternatively, the determination may be based on an inspection of the signals provided by I/O Component 150, e.g., confirming green light, verifying a predetermined audio signal is emitted, or the like.

[0094] In some exemplary embodiments, Wearable Device 120 may be utilized as a retail smart device. Wearable Device 120 may be configured to be worn by a shopper during self-service shopping, may be configured to be worn by a picker fulfilling an online order, by a retailer or an employee placing stock, or the like. As another example, the device may be worn by a cashier during checkout activity, such as to scan the products and create the digital shopping list. Additionally or alternatively, Wearable Device 120 may be utilized for other tasks, such as safety monitoring, actions monitoring, logging user actions, augmented reality games, virtual reality applications, or the like.

[0095] In some exemplary embodiments, Wearable Device 120 may be configured to continuously monitor Hand 130 between a check-in and check-out activities. Such monitoring may comprise obtaining and analyzing input related to Hand 130, such as visual input, geospatial location, or the like.

[0096] In some exemplary embodiments, Sensor 110 may be configured to capture at least an Interior Portion 132 of Hand 130. Interior Portion 132 may comprise Distal Portion 134 of a Palm 133. Sensor 110 may be configured to face Palm 133 whereby capturing Distal Portion 134. The visual input may capture at least a portion of the object when the object is being held by Hand 130, such as when being grasped by Fingers 136 of Hand 130, or the like. At least a portion of the visual input of Sensor 110, such as about 5%, about 10%, about 50%, may comprise a view of Interior Portion 133 to enable identification of the object.

[0097] It may be noted that a view of Sensor 110 may be blocked, at least in part, by Hand 130, or by the object held by Hand 130. As a result, Sensor 110 may not be enabled to capture the whole environment surrounding Hand 130, such as the face of the user, other people in the surrounding environment, unrelated objects, or the like. Additionally or alternatively, the view of Sensor 110 may be a spherical view capturing 360 degree panoramic space surrounding Hand 130. In some exemplary embodiments, the spherical view may have a relatively limited view, such as a spherical view with a radius of up to about 10 centimeters around Hand 130, up to about 25 centimeters around Hand 130, or the like.

[0098] In some exemplary embodiments, Sensor 110 may be positioned on a protrusion of Wearable Device 120, distancing Sensor 110 from the surface of Hand 130. Such placement may be useful for preventing the view to be blocked by the base of the Palm 133.

[0099] In some exemplary embodiments, Wearable Device 120 may be configured to provide images captured by Sensor 110 to be utilized by Analysis Component 160. Analysis Component 160 may be configured to identify an action performed by Hand 130 and to identify an object upon which the action is performed. Additionally or alternatively, Analysis Component 160 may be configured to identify or distinguish between a foreground and a background of the visual input. Analysis Component 160 may be embedded within Wearable Device 120 (as exemplified herein) or may be located on a device external thereto (not shown), such as on a server, a backend device, or the like. [0100] Additionally or alternatively, Communication Component 140 configured to connect Wearable Device 120 to a controller (not shown) external to Wearable Device 120. Analysis Component 160 may be connected to or embedded in the controller. The controller may be configured to determine a responsive action based on the action or the item. The responsive action may be associated with the purpose of monitoring actions of Hand 130, such as reporting the action or the object, calculating a check based on the action and the object, issuing an alert based on the action or the object, or the like.

[0101] In some exemplary embodiments, Wearable Device 120 may be devoid of a deactivation interface for the user. Activation and de-activation of Wearable Device 120 may be performed automatically by the controller. In some exemplary embodiments, power source (not shown) of Wearable Device 120, such as battery, may be sealed and the subject may not have access thereto. Additionally or alternatively, Wearable Device 120 may be provided with a limited de-activation interface for the user, that enables the user to de-activate Wearable Device 120 upon finishing a shopping session, based on permission from the controller, or the like.

[0102] In some exemplary embodiments, Wearable Device 120 may be configured to be utilized for self-service shopping. Wearable Device 120 may be configured to be utilized to identify items grabbed by Hand 130 and moved to or from a physical shopping tote of the user, wherein the items are identifiable based on input of Sensor 110. Wearable Device 120 may be configured to be associated with a virtual cart upon initiating a selfshopping session. The virtual cart may indicate a list of items shopped by the user. The virtual cart may be automatically updated based on items moved to and from the shopping cart by Hand 130. In some exemplary embodiments, Wearable Device 120 may comprise a tampering detection module (not shown) that is configured to monitor and detect a tamper event during a shopping session of the user, avoid monitoring user activity outside the shopping session, or the like.

[0103] Additionally or alternatively, Wearable Device 120 may be configured to be utilized for manual fulfillment of a shopping order of a customer. The shopping order may comprise a list of items. Hand 130 may be of a picker tasked with picking items to fulfill the shopping order of the customer. Wearable Device 120 may be configured to identify actions of picking up an object by Hand 130 and placing the object in a tote associated with the shopping order of the customer. [0104] Additionally or alternatively, Wearable Device 120 may be configured to be utilized for protecting the user or other related subjects. The responsive action determined based on the input of Sensor 110 may comprise comparing the action performed by Hand 130 with a safety rule. In response to a violation of the safety rule, a safety alert may be issued. Additionally or alternatively, Wearable Device 120 may be configured to be utilized for monitoring a health-care system. Wearable Device 120 may be configured to continuously monitor the hand of health-care workers during treatment of patients.

[0105] In some exemplary embodiments, other embodiments of the wearable device may be utilized, such as a device configured to be worn on the chest of the user, embedded in a vest to be worn by the user, a hat shaped device configured to be worn on the head of the user, a device configured to be worn on the forehead of the user such as using elasticized straps, or the like. Such wearable devices may also comprise visual sensors (such as in Sensor 110) configured to capture at least an interior portion of the hand of the user, objects being held by the hand user actions performed by the hands of the user, or the like.

[0106] Referring now to Figure 2 showing a schematic illustration of an exemplary self- service shopping or online- shopping fulfillment for customers scheme, in accordance with some exemplary embodiments of the disclosed subject matter.

[0107] In some exemplary embodiments, a Retail Store 200 may provide a self-service shopping, using smart wearable devices, such as Wearable Device 120. It is noted that all functionalities of the smart wearable devices may be based on on-device computations or on off-device computations, such as performed by an edge device in Store 200, a remote server, a cloud-based server, or the like.

[0108] In some exemplary embodiments, when a Customer 210 enters Store 200, or checks into a system of Store 200 (such as by using a Mobile Device 212 or otherwise), Customer 210 may receive one or more retail smart wristbands, such as Wearable Device 214, from Store 200. In some exemplary embodiments, Store 200 may have a designated location (205) where wearable devices are placed and await to be picked up by customers, similarly to the location of the Available Carts 206. In some exemplary embodiments, in Location 205, the wearable devices may be charged, to ensure they have sufficient power level. Wearable Device 214 may be affiliated or assigned to Customer 210 for the duration of the shopping session. Customer 210 may be requested to pair or synchronize Wearable Device 214 to a device thereof, such as through a designated mobile app of Mobile Device 212, by registering to an account associated with Store 200, by scanning Wearable Device 214 using a Scanner 201, or the like. After the relevant pairing, each action performed with the hand(s) of Customer 210 wearing Wearable Device 214 may be attributed to the account of Customer 210. In some exemplary embodiments, the check-in activity may be performed manually by a retailer or a manger of Store 200, upon providing Wearable Device 214 to Customer 210, in response to Customer 210 removing Wearable Device 214 from a docketing station thereof (e.g., 205), upon an activity of Customer 210 related to providing a means of payment, or the like.

[0109] In some exemplary embodiments, Customer 210 may be continuously monitored in Store 200 using Wearable Device 214. In some exemplary embodiments, Wearable Device 214 may be configured to monitor hand actions of Customer 210. In some exemplary embodiments, Wearable Device 214 may be synchronized with other monitoring devices of Store 200, such as security cameras (201), scales, or the like.

[0110] Additionally or alternatively, the wearable device may be a personal device of the shopper, such as Wearable Device 222 of Customer 220. Wearable Device 222 may be configured to connect to a system of Store 200. The check-in may be performed automatically when Customer 220 enters the store, when Wearable Device 222 connects to a monitoring device Store 200, or the like. It may be noted that in some cases a preliminary registration prior to a first check-in activity may be required, such as to update a shopping profile of Customer 220, creating an account, providing payment method, providing a shopping list, or the like. Customer 220, may utilize his personal Wearable Device 222 to perform self-service shopping in Store 200. When Customer 220 enters Store 200 while wearing Wearable Device 222, Wearable Device 222 may be configured to recognize that Customer 220 has entered Store 200, and may synchronize with the store's system. As an example, Wearable Device 222 may connect to the Wi-Fi of Store 200, and accordingly recognize Store 200. As another example, Wearable Device 222 may be preconfigured to recognize Store 200 and connect to systems thereof. As yet another example, Wearable Device 222 may be manually paired with systems of Store 200, such as by scanning an identifier thereof using Scanner 201, pairing with a shopping cart from Available Carts 206, or the like. As yet another example, Wearable Device 222 may be paired to a mobile device of Customer 220, which may be utilized to manually check in Store 200, may connect via a respective mobile app, or the like. Additionally or alternatively, a geo-fence associated with Store 200 may be used to detect entering and exiting Store 200. Wearable Device 222 may be configured with geo-fence or locationbased activation. Wearable Device 222 may be activated when Customer 220 enters Store 200, by using geo-fence feature or other signals to a mobile app, or directly to Wearable Device 222, which may activate Wearable Device 222. It may be noted that as Wearable Device 222 is the shopper’s property (e.g., not received in the course of check-in to Store 200), the process of check in may activate or turn on Wearable Device 222, and then each action performed with the hands wearing Wearable Device 222 within the store will be attributed to Customer 200. Furthermore, Customer 220 may utilize a personal Shopping Bag 224 instead of a shopping cart of Store 200, for environmental purposes, for convenience, in order to perform faster shopping, or the like.

[0111] In some exemplary embodiments, the disclosed subject matter may be utilized for instore pickers performing online- shopping fulfillment for customers in Store 200. The instore pickers, such as Picker 280, Picker 240 may be employees of Store 200, that are directly engaged with or has any other direct relationship with Store 200. Additionally or alternatively, the instore pickers, such as Picker 230 may be independent pickers, pickers that are not directly employed by Store 200, or the like. The instore picker may implement the picking tasks that the store (or the customer or a third party, as for Picker 230) tasked her with. In some exemplary embodiments, the picking task may be separated from the delivery task, both performed in order to fulfill an order, e.g., online order, of a direct customer of Store 200.

[0112] In some exemplary embodiments, each picker may be assigned with a wearable device (232, 242 and 282). In some exemplary embodiments, there may be several advantages to using Wearable Devices 232, 242, 282, or the like, for instore pickers 230, 240 and 280, such as avoiding errors, avoiding shoplifting, improving efficiency, or the like. In some exemplary embodiments, Wearable Devices 232, 242, 282 may be utilized in verifying that the right items are put in the correct shopping cart, thus assisting in avoiding errors. Additionally, or alternatively, all items picked are automatically registered at the shopping list, and the risk of shoplifting may be reduced. Additionally, or alternatively, Wearable Devices 232, 242, 282 may improve efficiency of the instore pickers 230, 240 and 280, such as due to avoiding the need of manually scanning picked items.

[0113] In some exemplary embodiments, each wearable device may be utilized to continuously monitor actions of the shopper or picker wearing the wearable device during the shopping session. The monitoring may be performed continuously between the checkin activity and a respective checkout activity. In some exemplary embodiments, Wearable Device 222 may be configured to provide information to the owner of Store 200 only with respect to the shopping session within Store 200. As an example, if Customer 220 uses Wearable Device 222 in several stores, each store may gain an access to information relating to visiting the respective store only and not to information relating to other stores. In some exemplary embodiments, analytics and general information may be tracked by a general service provider, who may not necessarily be affiliated to any specific store.

[0114] In some exemplary embodiments, a Picker 240 may be wearing a Wearable Device 242 during fulfilment of a customer order in Store 200. Sensors of Wearable Device 242 may be configured to capture at least an interior portion of the hand of Picker 240 wearing Wearable Device 242, and provide visual input thereof. In some exemplary embodiments, a Picker 240 may utilize Wearable Device 242 for manual fulfillment of shopping orders of customers, such as on-line orders. Each customer may provide a list of items to be purchased. The list of order may be viewed to Picker 240 via a screen, such as on a Computing Device 243 managing such orders. Picker 280 may fulfill two or more different orders simultaneously, thus improving the overall number of items Picker 280 can handle per timeframe. Picker 240 may pick items listed in the list of items to fulfill the shopping order of the customer. Wearable Device 242 may be configured to identify when Picker 240 picks up an object and place it in a tote associated with the shopping order of the customer. In response to identifying a corresponding item to the object in the list of items, such corresponding item may be automatically marked as fulfilled.

[0115] Wearable Device 242 may be configured to identify each item that Picker 240 performs an action with (such as picking, holding, putting in a shopping bag or cart, or the like), based on the visual input. As an example, Wearable Device 242 (or an associated software thereof, on-device or in a back-end) may be configured to identify that Picker 240 picks up Object 245, and place it in Shopping Bag 246. Additionally or alternatively, Wearable Device 242 may be configured to utilize additional types of input to identify the object and the action, such as positioning readings of Wearable Device 242, accelerometer readings of Wearable Device 242, or the like. In response to detecting the object and the action performed thereon by Picker 240, Wearable Device 242 (or the associated software thereof) may be configured to update a virtual cart of Picker 240 to include Object 245. Additionally or alternatively, Wearable Device 242 may be configured to determine, such as based on a series of visual inputs over time, that Picker 240 has returned Object 245. As an example, the context of Object 245 may be determined based on the background of the visual input, thus, determining that Picker 240 removed Object 245 from Shopping Bag 246, placed Object 245 back to its location, to another location in Store 200, or the like. In response to such determination, Wearable Device 242 (or the associated software thereof) may be configured to update the virtual cart of the customer order fulfilled by Picker 240 to exclude Object 245. In some exemplary embodiments, the virtual cart may be displayed for the customer, who may be enabled to communicate with Picker 240, such as using Mobile Device 244, or the like. Additionally or alternatively, the virtual cart may be retained by systems of Store 200 and sent to the customer upon finishing the shopping session. Additionally or alternatively, Wearable Device 242 may be configured to emit an auditory cue such as a beep, or a visual output such as a green light, or the like, indicating the addition (or removal) of an object to the virtual cart.

[0116] In some exemplary embodiments, using the data gathered by the devices within Store 200, a virtual map of Store 200 may be created and utilized for designing efficient picking route within the store for a single shopping cart, or for multiple shopping carts that are picked simultaneously. As an example, the items appearing in the aggregated picking tasks from a single order or from a multiplicity of orders may be identified, the location of each item in the store may be identified in the virtual map and a shortest route passing through all the locations may be identified and displayed to the instore picker. In some exemplary embodiments, there may be alternative locations for a same item, such as placed in two locations (e.g., in a specific aisle and next to the checkout line; or in its designated location and in a place where a customer left an unwanted item). The shortest route may be determined by selecting between the two alternatives.

[0117] In some exemplary embodiments, Picker 230 may be a third-party picker who may not be engaged directly with Store 200, e.g., not a store worker. As an example, the consumer may order items (e.g., using cross-store Stock Keeping Units (SKUs)) from a picking entity and pays that entity directly. Picking entity may be independent of the store from which the item is picked, may fulfill orders by picking in several stores, or the like. In some exemplary embodiments, Picker 230 may perform the picking task on behalf of the picking entity. Picker 230 may utilize Wearable Device 232 to facilitate self-checkout session to reduce fulfillment time, where applicable. The disclosed subject matter may enable crowd-source picking model. Any person may register to perform a picking task without material prior training and with the ability of the picking company to supervise the picker action and performance.

[0118] In some exemplary embodiments, Picker 230 may fulfill multiple orders simultaneously, one after the other, or the like. Picker 230 may obtain multiple shopping orders of different customers. The items of each shopping order may be picked and placed in a tote associated with the customer. As an example, Cart 236 may comprise three different totes (237, 238 and 239) each of which utilized to pick a shopping order of a different customer. Picker 230 may obtain a combined list of items sorted according to their location in Store 200 to enable faster collecting of the items. Each item in the combined list may be marked to indicate the relevant customer. Wearable Device 232 may be configured to monitor Picker 230 while fulfilling each shopping order. Picker 230 may be enabled to configure Wearable Device 232 to the relevant customer whenever switching between the orders. Additionally or alternatively, Wearable Device 232 may be configured to identify to which customer the item belongs based on identifying the tote in which the item is placed. Wearable Device 232 may be configured to provide visual input capturing the tote that the item is being placed therein, an identifier thereof, such as a barcode, an identifying color, or the like.

[0119] In some exemplary embodiments, Picker 230 may perform the picking and delivery tasks. In some exemplary embodiments, Wearable Device 232 may be configured for out-of-store monitoring, instead of or in addition to instore monitoring. In some exemplary embodiments, out-of-store monitoring may be utilized to verify that the items are not taken by the picker from the shopping baskets, to verify that the items are actually delivered to the consumer. In some exemplary embodiments, instore monitoring may be performed without cooperation of the store itself and while Picker 230 may require visiting a check-out station, which may be manned. In other cases, Picker 230 may utilize a self-checkout system of Store 200 which may be independent of Wearable Device 232, such as using other systems for performing self-checkout. Additionally, or alternatively, Wearable Device 232 used by Picker 230 may integrate with the systems of Store 200 and may facilitate self-checkout in Store 200 itself.

[0120] In some exemplary embodiments, out-of-store monitoring can be done by continuous recording of the hands vision out of Store 200 and storing them locally in the wristbands memory card and transmitting that recording when the wristband is connected to Wi-Fi, or real time transmission by using embedded SIM card, or other network connectivity means. In some exemplary embodiments, the wristband may recognize when a Wi-Fi signal is lost and automatically switches to saving data on local memory or transmitting to cellular network, if available. In some exemplary embodiments, the monitoring may begin when the picker commences to perform a picking task and may continue even with the picker exists the store and until her task is completed. For example, in case the task ends when the goods are delivered to the consumer, the monitoring may end at that time. In some exemplary embodiments, an image of the shopping cart that is placed at the delivery location (e.g., outside the door of the consumer, at a pad from which a drone picks the goods, or the like) may be captured and used as proof of delivery, in case of a dispute. The information gathered during the monitoring may be utilized for automatic dispute resolution, such as based on the location of the picker during performance of the task (e.g., indicating potential theft), based on accelerometer and gyroscope reading which may indicate an attempt to remove items from the tout delivery to the consumer, or the like.

[0121] Additionally or alternatively, the wearable device such as Wearable Device 282 may be utilized by the retailer or workers of Store 200, such as Worker 280, for different purposes, such as arranging Store 200, stocktaking, mapping locations of objects within Store 200, determining the exact location of each item in Store 200, inventory checking, verification of the quantities and condition of items in Store 200, mapping shelfs in Store 200, or the like. Wearable Device 282 worn by Worker 280, may be configured to follow each object being held by Worker 280 from its position at a Delivery Box 285 to the shelves of Store 200. Visual input from different wearable devices utilized in Store 200, or other sensors monitoring Store 200, such as Camera 201, may be matched to enable the system to draw the structure Store 200. After the system learns the structure of Store 200, the system may be configured to map the location of each item put by Worker 280 on the shelves. The system may have a virtual map of the entire Store 200 together with the items put on their shelves or arranged elsewhere in Store 200. Wearable Device 282 may be configured to identify an action of placing an Object 284. In response, Wearable Device 282 may be configured to determine a geospatial location of Object 284 after being placed, and updating a mapping of items in Store 200 to indicate that Object 284 is located at the geospatial location. Additionally or alternatively, when a shopper takes an item from a shelve, the wearable device of the shopper may be configured to recognize the shelf (such as when the shopper wrist is close to that shelf). Such feature may assist with identifying the item, which the system has prior knowledge of its location on that shelf, from monitoring the store arrangement, or the like. Mapping of the shelves may be supported by unique identifiers (such as stickers) which may be pasted on, or otherwise affixed to the shelves' fronts. The shelves may be marked such as using a 2D mapping of aisle number and shelf number in the aisle. Additionally or alternatively, each shelf may be divided into cells creating a 3D mapping. Each shelf or shelf cell may have a unique identifier (such as combination of letters and numbers) which may ease the process of Wearable Device 282 in mapping Store 200 and further recognition of the exact location of Wearable Device 282 when used by Worker 280, or other wearable devices worn by shoppers. In some exemplary embodiments, the identifier may provide an approximated location. The approximated location may be of size of about 1 meter x 1 meter x 1 meter, or the like. Additionally or alternatively, the approximated location may be of size of about 80 cm in width, 30 cm in depth and 30 cm in height. The measurements may be based on the size of the shelf, such as the height of the shelf, the depth of the shelf, or the like. In some exemplary embodiments, the approximated location may be utilized to reduce complexity of identifying the item. For example, based on the approximated location, potential items that are retained in the approximated location (or nearby locations) may be known and may be used as the "immediate suspects" for matching when the shopper picks up an item. Determining whether the picked up item is a specific item is potentially an easier computational task than attempting to identify which item it is when compared to a database of thousands of items. In some exemplary embodiments, matching may be performed with respect to a first repository having a small number of items that are located nearby, and with respect to a second repository having all potential items in the store. A reduced confidence level may be sufficient for matching an item in the first repository in comparison to the minimal confidence threshold required for the second repository. Additionally or alternatively, the matching process may take into account the image together with the distance between the item and the approximated location, such as increasing likelihood of matching when the item is stored nearby the approximated location. In some exemplary embodiments, items that are usually retained in one place may be naturally moved by shoppers to other places that are still nearby.

[0122] In some exemplary embodiments, a database or a catalog of items may be retained by the system managing the self-service shopping using the wearable devices. Prior to activating the solution, all items which may be sold in Store 200 may be pictured from different angels, categorized and stored in a designated database, e.g., a catalog database. When the system (through a wearable device) recognizes that the hand(s) made an action or got close to an object, the picture, video or signal of the item may be matched with the database, and identified thereof (or not identified, if such object is not listed in the database or matching has not succeeded). Additionally or alternatively, a partial database may be utilized for each store. The partial database may comprise items that are known to be in the store in a certain location. In order to speed the wearable device's identification of an object, instead of searching the entire database each time, only the items, which the service is aware of being located in the location of the wearable device, baes on positioning readings thereof, may be searched. In case that the system does not find the object in that partial database, the system may search the whole database for that object. The system may be configured to know what items are located in Store 200 based on an inventory list obtained from different wearable devices worn by customers, from Wearable Device 282, from the relevant Point of Sale (PoS) used at the store, other solution managing the store’ s inventory, or the like. Each item that is located in Store 200 may be identified and listed in the partial database of the store.

[0123] In some exemplary embodiments, the system utilizing the wearable devices may be configured to learn shoppers’ behavior in general, such as the ways and methods for choosing, picking, holding, moving, releasing items, or the like, the unique way of each shopper to perform these actions, or the like. Such learning may be performed using machine learning or other techniques. Learning shoppers’ behavior may reduce false signals or portion of undefined shopper's actions. Additionally or alternatively, the wearable device may be configured to learn properties of the items, such as shape, from different angles, and improve the identification of the items to minimize false or nonidentifications.

[0124] In some exemplary embodiments, there may be multiple of customers in Store 200. Some of which may utilize wearable devices, and some may not. In some exemplary embodiments, some of the customers may conduct self-service shopping while other may conduct traditional shopping which also comprise manual scanning of the items, e.g., by a cashier during check-out. [0125] Referring now to Figure 3 showing schematic illustrations of visual inputs provided by a hand action monitoring wearable device, in accordance with some exemplary embodiments of the disclosed subject matter.

[0126] In some exemplary embodiments, a wearable device, such as 120 depicted in Figure 1, may be worn on a Hand 310 of a subject. The wearable device may comprise one or more sensors configured to capture at least an in interior portion of Hand 310. In some exemplary embodiments, the one or more sensors may be configured to be placed in a location and orientation in the wearable device enabling monitoring of Hand 310. The one or more sensors may be configured to act as a SnorriCam, a POV camera, or the like, while Hand 310 appears in a fixed position in the center of the frame to enable monitoring activity thereof. Additionally or alternatively, the wearable device may be configured to be worn on a wrist of Hand 310, whereby positioning the one or more sensors to face a palm of Hand 310, or at least a distal portion thereof, such that enabling to capture at least a portion of an object when the object is being held by Hand 310, or when being grasped by fingers of Hand 310, or the like.

[0127] In some exemplary embodiments, the one or more sensors (herein also referred to as the sensors) may be configured to provide visual input that at least a portion thereof comprises a view of the interior part of Hand 310, such as Images 301, 302, 303 and 304. As an example, Image 301 captures a view of Hand 310 in front of a Shelf 350 within the store, with portion of the objects on the shelf, such as Object 320. As another example, Image 302 captures a view of Hand 310 placing Object 320 in a Basket 340. As yet another example, Image 303 captures Hand 310 along with a portion of Basket 340, that comprises other objects picked by the user, such as Object 355. As yet another example, Image 304 captures a different view of Hand 310 being free from any object, while approaching other items in the store such as Object 360. It may be noted that different images may capture different views associated with Hand 310. However, at least a portion of the image (such as about 5%, about 10%, about 50%, or the like) may comprise a view of the portion of Hand 310. Such portion may vary from one image to another, based on the angle of Hand 310, the position thereof, the action being performed thereby, or the like. As an example, Image 301 captures a smaller portion of Hand 310 comparing to Image 302.

[0128]

[0129] [0130] In some exemplary embodiments, Images 301-304 may be a series of subsequent visual inputs obtained from the one or more sensors one after another. The series pf subsequent visual inputs may comprise a first visual input (e.g., Image 301), a second visual input (e.g., Image 302), a third visual input (e.g., Image 303) and a fourth visual input (e.g., Image 304). Image 301 may precede Image 302 and Image 303 may precede Image 304. The timeframe between each image and its successive image may be constant, predetermined, such as about no more than 1 second, no more than 500 milliseconds, no more than 200 milliseconds, no more than 100 milliseconds, no more than 50 milliseconds, or the like. Additionally or alternatively, the each successive image may be configured to capture the next frame or series of frames, such as no more than 30 frames, no more than 10 frames, no more than 5 frames, no more than a single frame, or the like.

[0131] In some exemplary embodiments, the view of the sensor may be blocked, at least in part, by Hand 310. As an example, in Image 302 and Image 303, the view of the sensor is limited to a portion of Basket 340, without showing the external environment thereof.

[0132] In some exemplary embodiments, a pick action by Hand 310 with respect to Object 302 may be automatically identified based on the Image 301 and Image 302. The automatic identification of the pick action may be performed based on a determination that in Image 301 Hand not touching the object (320) and based on a determination that in Image 302, Hand 310 touches Object 320. Additionally or alternatively, a release action by Hand 310 with respect to Object 320 may be performed based on Image 303 and Image 304. The automatic identification of the release action may be performed based on a determination that in Image 303 Hand 310 touches Object 320 and based on a determination that in Image 304 Hand 310 does not touch Object 320.

[0133] In some exemplary embodiments, a digital mapping of objects to locations may be updated automatically based on the automatic identification of the pick action and/or the release action. The digital mapping may comprise updating a location of a digital representation of Object 320 from a first location (e.g., Shelf 350) to a second location (e.g., Basket 340 or Shelf 360).

[0134] In some exemplary embodiments, an identification of the action performed by Hand 310 may be performed based on the series of the subsequent visual inputs, or a portion thereof, such as based on two or more images. In some exemplary embodiments, the wearable device may be configured to recognize when Hand 310 is getting close to an item (such as Object 320), picking, holding (e.g., the object remains being held by Hand 310), moving it (e.g., background picture changed), releasing an item, or the like. The identification of Object 320 and the actions performed thereon may be identified based on the visual input, such as Images 301-304.

[0135] In some exemplary embodiments, each visual input (e.g., an image such as Images 301-304) may comprise a foreground and a background. The foreground and the background of each image may be determined with respect to Hand 310. A determination that Hand 310 is holding an object (such as Object 320 in Image 302) may be performed based on an identification of the object in a foreground of the visual input. As an example, Object 320 may be identified in the foreground of Image 302, and thus determined to be the object of interest. In some exemplary embodiments, a recognition that an object is in a foreground of a visual input may be performed based on a movement pattern of Hand 310 in the series of visual inputs. As an example, Object 320 is determined to be in the foreground of Image 302 based on identifying the Hand 310 is moving towards Object 320 in Image 301 and/or opposite from Object 320 in Image 303, that Hand 310 is reaching Object 320 from different directions, or the like. Additionally or alternatively, the recognition that an object is in a foreground of a visual input may be performed based on an appearance pattern of the object within the series of visual inputs. As an example, the object being in a certain relative location with respect to Hand 310, the object being close to the palm of Hand 310, or the like. Additionally or alternatively, the background of the visual input may comprise items or objects that do not move with Hand 310. In some exemplary embodiments, image analysis techniques may be applied to identify Object 320. Object 320 may be recognized based on a catalog of items of the store, such as by comparing portions of Object 320 with the catalog of items.

[0136] In some exemplary embodiments, the action performed by Hand 310 with respect to Object 320 may be automatically identified based on a difference between Image 302 and the preceding visual input thereof, e.g., Image 301, or based on a difference between Image 302 and the successive visual input thereof, e.g., Image 303. As an example, based on Image 301, the wearable device may be configured to recognize that Hand 310 is about to perform an action on one of the items placed on Shelf appearing in the background of Image 301, such as Object 320, based on identifying that Hand 310 getting close to Object 320, and picking it. Based on Image 302, it may be identified that Hand 310 is holding Object 320 and moving it towards Basket 340. Based on Image 304, it may be identified that Hand 310 is releasing Object 320 and putting it in Basket 340. In Image 304, Hand 310 may be identified to be free again and ready to pick another item. Accordingly, a "picking action" of Object 320 from Shelf 350 to Basket 340 may be determined based on the series of visual inputs of Images 301-304.

[0137] Additionally or alternatively, features from the backgrounds of Images 301-304 may be utilized to identify other properties of the action being performed, such as information about the item being picked, the customer order associated with the picking action, matching between the item and an order list, or the like. As an example, additional information may be determined about Object 320, such as the type, category, name, shape, weight, size, price, or the like, based on the location of Object 320 prior to being held by Hand 310, as can be learned from Image 301.

[0138] Additionally or alternatively, other sensor readings from the wearable device may be utilized in order to identify the object or the action being performed thereon. As an example, a positioning reading of the wearable device, indicative of the location thereof, may be obtained, such as using a location sensor thereon, a location system of a device associated therewith, or the like. A subset of a catalog of items of the store may be determined based on the location, such as based on an input from the store, or the like. The subset of the catalog may comprise items located on Shelf 350, items located in the fridge comprising Shelf 350, diary items, or the like. A product recognition may be performed to identify Object 320 with respect to the subset of the catalog of items. As another example, according to accelerometer readings of the wearable device, a pattern of accelerations associated with a certain action may be determined. As yet another example, according to positioning readings, a determination that a location of Object 320 is changed such as from Shelf 350 to other location.

[0139] In some exemplary embodiments, according to Image 303, a context of the action and/or the object may be determined based on a background of the visual input. As an example, Additionally or alternatively, a first context of Object 320 associated with a first location of Object 320 may be determined based on a background Image 301 capturing Shelf 350, or other similar objects indicative of other properties of Object 320. A second context of Object 320 associated with a second location to which Object 320 is being moved to may be determined based on Image 302 capturing Shopping Bag 340 in the background. As another example, other items identifying the customer may be determined based on the background of Image 303 capturing the content of Shopping Bag 340, or the like. The background of Image 302, may be utilize to validate the virtual cart associated with the customer. The content of Shopping Bag 340 may be identified based on the images, and compared to the virtual cart being updated during the shopping session. In response to determining a discrepancy between content of the virtual cart and content of Shopping Bag 340, the virtual cart may be invalidated, updated based on the images, or the like. As an example, Object 355 may be identified in Image 303, while not be listed in the virtual cart. Accordingly, the virtual cart may be updated to include Object 355 or an identifier thereof.

[0140] In some exemplary embodiments, Image 301 may precede Image 302 by no more than a first predetermined threshold. Image 302 may precede Image 304 by no more than a second predetermined threshold. The first predetermined threshold and the second predetermined threshold are identical. The first predetermined threshold and/or the second predetermined threshold are selected from a group consisting of: no more than 1 second, no more than 500 milliseconds, no more than 200 milliseconds, no more than 100 milliseconds, no more than 50 milliseconds, no more than 30 frames, no more than 10 frames, no more than 5 frames, and no more than a single frame.

[0141] In some exemplary embodiments, the background of the visual input may comprise portions of the store in which the shopping session is being performed. As an example, Image 301 captures a Shelf 350 and Image 304 captures Shelf 360. Such images may be analyzed to recognize the shelf, such as based on an identifier thereof, a sticker pasted thereon, based on a prior knowledge of the location of the associated object on that shelf, from monitoring the store arrangement, or the like. As an example, based on prior knowledge of the location of Object 320, Shelf 350 may be identified. Additional action may be performed based on identifying the shelves, such as updating inventory of the respective store to indicate that Object 320 is purchased, maintaining the mapping of objects in the store, extracting additional information related to Object 320, such as offers or sales, expiration date, temperature (based on the type of the shelf, the shelf being in a refrigerator or a freezer, or the like), or the like.

[0142] Referring now to Figure 4 A showing a flowchart diagram of a method, in accordance with some exemplary embodiments of the disclosed subject matter.

[0143] On Step 410, a first visual input and a second visual input may be obtained from a sensor located on a wearable device. In some exemplary embodiments, the wearable device may be designed to be worn by a subject on a hand of the subject, such as Wearable Device 120 in Figure 1. The sensor may be configured to be placed in a location and orientation enabling monitoring of hand activity of the subject. The wearable device may be utilized to monitor the subject, and particularly, hand actions of the subject, during self-service shopping or a manual fulfillment of online- shopping for customers by the subject, in a store, such as depicted in Figure 2.

[0144] It may be noted that the first and the second input may be consequent visual inputs, e.g., visual inputs obtained from the sensor after a predetermined timeframe, such as every 10 milliseconds, every 100 milliseconds, every 500 milliseconds, every 1 second, or the like. As an example, the first and the second visual inputs may be two subsequent visual inputs from Images 301-304 of Figure 3.

[0145] In some exemplary embodiments, the subject may be a picker tasked with picking items to fulfill an order of a customer.

[0146] On Step 420, an object may be identified in a foreground of the first visual input. In some exemplary embodiments, the foreground and the background of the visual input may be with respect to hand of the subject. The background is any portion of the visual input that does not move with the hand of the subject. As an example, according to the location of the sensor in the wearable device, the hand of the subject may be configured to appear in a fixed position in the center of the frame. As another example, the visual sensors may be configured to present a dynamic, disorienting point of view from perspective of palm of the hand. The foreground of the visual input may be determined according to relativeness to the hand in the visual input. Additionally or alternatively, an identification that the object the foreground of the visual input may be performed according to a movement pattern of the hand, such as the hand moving towards the object, the hand moving opposite from the object, the hand reaching the object from different directions, or the like. Additionally or alternatively, an identification that the object the foreground of the visual input may be performed according to an appearance pattern of the object within the visual input, such as being in a relatively constant location in two consequent visual inputs, being in a constant location with respect to the hand, or the like.

[0147] In some exemplary embodiments, the first visual input may be analyzed to automatically determine a real-time shopping cart utilized by the subject during the shopping in the store. The real-time shopping cart may be a physical cart, a shopping bag, a personal bag, a tote, one of the hands of the subject, or any other container. The realtime shopping cart may change during the shopping session. As an example, the subject may utilize different shopping bags for different types of objects, the subject may keep one or more objects in her hands before moving to the shopping cart, the subject may put one or more objects in a plastic bag before being places in the shopping cart, or the like. The real-time shopping cart may be determined to be a portion of the background of the visual input, identification thereof may be utilized to identify the foreground of the first visual input.

[0148] In some exemplary embodiments, a subset of a catalog of items may be determined based on the positioning reading of the wearable device, such as a catalog comprising items located in a respective location within the store. A product recognition with respect to the subset of the catalog of items, may be performed to identify the object.

[0149] In some exemplary embodiments, in case of a misidentification of an item, the disclosed subject matter may be configured to retain a list of unresolved items. In some exemplary embodiments, as long as there are unresolved items, the cart may be considered invalidated. In some exemplary embodiments, an unresolved item may be added in view of a misidentification of an item (e.g., it may appear to be any one of several potential items; identification is below a predetermined confident threshold, or the like). Additionally or alternatively, an unresolved item may be added in view of a required additional action. As an example, in case of items sold in bulks, such as fruits and vegetables, additional information may be required to be entered, such as the weight of the collected goods, the goods cost, a barcode identifier (e.g., distinguishing between apples and organically grown apples; a barcode indicating cost and/or weight), or the like. In some exemplary embodiments, open items may be resolved automatically at a later time, such as using additional data gathered since the initial identification of the open item issue. As an example, images captured minutes after the item was first identified and placed in the shopping cart, may shed light on which item was taken, such as due to capturing a different view of the item (e.g., showing the name of the item clearly, showing a barcode of the item, or the like). In some exemplary embodiments, other obtained items may also be useful in resolving open issues - for example, if the user has a shopping list that indicates purchasing both diet coke and regular coke bottles, if a bottle remains unresolved, the fact that the user picks up another bottle that is matched to be regular coke bottle, increases the likelihood and the confidence in the determination that the unresolved item is in fact a diet coke bottle.

[0150] Additionally or alternatively, in case of a misidentification of an item, the disclosed subject matter may be resolved by identification of the background of the item. For example, if the item may appear to be any one of several potential items, if it is found in location X, its identification is resolved. In some exemplary embodiments, some actions may be pre-conditioned on the lack of any potential issues, such as invalidated cart, remaining open items, or the like.

[0151] On Step 430, a determination that the subject is holding the object in the first visual input, may be performed. In some exemplary embodiments, the determination may be performed based on the identification of the object in the foreground of the first visual input. It may be noted that any action performed on the object by the hand of the subject comprises a phase in which the hand holds the subject. Such phase may be captured prior to identification of the object, after identification of the object, or the like.

[0152] On Step 440, an identification that the object is not in a foreground of the second visual input may be performed. In some exemplary embodiments, the identification that the object is not in the foreground may be based on the object not appearing at the second visual input at all. Additionally or alternatively, the identification that the object is not in the foreground may be based on the object being identified in the background of the second visual input, e.g., not moving with the hand of the subject.

[0153] On Step 450, a hand-based action of the subject with respect to the object may be automatically identified based on a difference between the first visual input and the second visual input.

[0154] In some exemplary embodiments, the action may be picking up an item for sale, holding an item, returning an item to its initial location, putting the item in the real-time shopping cart, removing the item form the real-time shopping cart, changing the realtime shopping cart, or the like. Additionally or alternatively, the action may be related to the fulfilment of the shopping order performed by the picker, such as picking up an object, holding an object, returning an object to its initial location, placing the object in a shopping cart or a tote associated with the shopping order of the customer, removing the object form the shopping cart, or the like.

[0155] As an example, identifying that the object is in the foreground of the first visual input and then not being in foreground of the second visual input may be indicative of a placement action. (Step 452).

[0156] An alternative scenario may be shown in Steps 424-454. [0157] On Step 424, an object may be identified in a foreground of the second visual input. In some exemplary embodiments, determining that the object appears in the foreground of the first visual input may be performed based a sequence of visual inputs that comprise the second visual input. Determining that the object appears in the foreground may comprises: identifying a pattern of appearance of the object in the sequence of visual inputs. As an example, the object appearing in constant relative location with respect to the hand, the object being covered by the hand, the object covering the visual input, or the like. Additionally or alternatively, determining that the object appears in the foreground may comprise: identifying a movement pattern of the hand of the subject with respect to the object in the sequence of visual inputs, such as a movement of the hand towards or from the object, or the like.

[0158] On Step 434, a determination that the subject is holding the object in the second visual input, may be performed. In some exemplary embodiments, the determination may be performed based on the identification of the object in the foreground of the second visual input.

[0159] On Step 444, an identification that the object is not in a foreground of the first visual input may be performed.

[0160] On Step 454, the hand-based action of the subject with respect to the object may be automatically identified to be a picking action based on identifying that the object is not in the foreground of the first visual input and then is in foreground of the second visual input.

[0161] On Step 490, a responsive action may be performed based on the identified hand-based action and the object.

[0162] In some exemplary embodiments, the responsive action may be a fulfillment- related action related to the picking of the custom order. The object may be identified to be associated with the order of the customer. In response to identifying that the picker picked up the object and placed the object in a tote associated with the order of the customer the fulfillment-related action may be performed (e.g., updating a virtual shopping cart, updating a check, or the like).

[0163] Additionally or alternatively, the responsive action may be an alert action. In response to identifying that the picker picked up the object and placed the object in a tote associated with the order of the customer; and identifying a mismatch between the object and a list of items in the order of the customer; the alert action may comprise issuing an alert to the picker indicating the mismatch.

[0164] In some exemplary embodiments, the responsive action may be related to the shopping process. The responsive action may comprise updating a virtual cart of the subject to include the object picked up by the subject as a purchased item. In some exemplary embodiments, a content of the virtual cart may be displayed to the subject, such as on a mobile device of the subject, on a screen on the shopping cart, or the like. The responsive action may further comprise highlighting the object as the recent item added to the cart, suggesting approval of the item by the subject, displaying the price of the object, alerting the subject of sales associated with the object, or the like. The responsive action may further comprise emitting an auditory cue indicating the addition of the object to the virtual cart, or the like. In some exemplary embodiments, the responsive action may further comprise automatically calculating an updated check to include the price of the object.

[0165] The responsive action may be related to the picking process (e.g. the fulfillment of the shopping order of the customer). In some exemplary embodiments, the responsive action may comprise identifying a corresponding item to the object in the list of items, and marking the corresponding item as fulfilled. Additionally or alternatively, the responsive action may comprise identifying a mismatch between the object and the list of items, and accordingly alerting the picker of the mismatch. In some exemplary embodiments, a check-out activity being performed in association with the wearable device, indicative of finishing the order of the associated customer may be detected. In some exemplary embodiments, the continuous monitoring may be terminated, in response to the check-out activity. Additionally or alternatively, the continuous monitoring may be continued with respect to other orders fulfilment of other customers. In some exemplary embodiments, a transaction may be performed based on content of the virtual cart of the customer. In some exemplary embodiments, a check may be calculated based on the items that were identified to be picked, put in the shopping cart and updated in the virtual cart.

[0166] In some exemplary embodiments, the wearable device may be configured to provide a real-time signal and feedback for the subject, a retailer, or the like. The realtime signal may comprise identification of the object and actions made with it. Such realtime signal may be a positive/negative signal, such as green light displayed by wearable device to make a positive identification of an item, and a red light which will be displayed when the retail smart wristband identifies that an item was picked but was unsuccessful with identifying the parameters of the item (unique name or identifier) or the action made with the item, or the like. Additionally or alternatively, the positive or negative identification may be displayed using the shopping cart of the subject, such as via the screen, using LED lights, or the like. The signal may be sent in real-time or later to the retailer as well, and may be used by the retailer, for example, in real time to determine if a shopping cart or a shopping bag should be “qualified” or “disqualified” for continuing the shopping under self-service. Additionally or alternatively, the objects which the system recognized as released in the shopping cart or bag, may be listed in a designated mobile app. The subject may be able to check in real-time if the system’s list is accurate, and to avoid the inconvenience caused by leaving the store with unlisted items.

[0167] Additionally or alternatively, in response to determining a returning action, the responsive action may comprise updating the virtual cart of the subject to exclude the object. The responsive action may further comprise emitting an auditory cue indicating the removal of the object to the virtual cart, updating the check to exclude the price of the object, suggesting alternative items to the subject, or the like.

[0168] Additionally or alternatively, in response to detecting a tampering event, the responsive action may comprise performing an anti-tampering action, such as issuing an alert, ending the shopping session, indicating the associated object as a purchased item, or the like. It may be noted that the wearable device is configured to perform detection of such tampering events, only during the self-service shopping session. The subject may be able to perform the tampering event after the self-service shopping session ends without resulting in the anti-tampering action.

[0169] Additionally or alternatively, the wearable device may be utilized to remove a theft detection tag coupled with the object. The responsive action may comprise indicating the object as purchased, ensuring payment for the object, or the like.

[0170] Referring now to Figure 4B showing a flowchart diagram of a method, in accordance with some exemplary embodiments of the disclosed subject matter.

[0171] Step 450b presents an alternative scenario to Step 450a shown in Figure 4A.

[0172] On Step 462, a first context of the object may be determined based on a background of the first visual input. In some exemplary embodiments, the first visual input provided by the wearable device may be enriched using input from sensors of the stores, such as by providing temperature information, weights of objects, or the like. Additionally or alternatively, the first visual input may be provided with additional input from other sensors located on the wearable device such as positioning reading of the wearable device, accelerometer readings of the wearable device, or the like.

[0173] On Step 464, a second context of the object may be determined based on a background of the second visual input. In some exemplary embodiments, the background of the second visual input may be totally or partially blocked by the object. In such cases, the second context of the object may be determined based on the background of the first visual input, a background of a successive visual input, the difference therebetween, or the like. As an example, the context may be determined based on the last identified physical location of the object, and the next identified physical location thereof. Additionally or alternatively, the second context may be determined based on additional sensory data obtained from other sensors of the wearable device, such as location information, accelerometry pattern, or the like.

[0174] On Step 466, the hand-based action of the subject with respect to the object may be identified based on a difference between the first context and the second context.

[0175] In some exemplary embodiments, the difference between the first context and the second context may be indicative of a change in a physical location of the object. The hand-based action may be identified based on the change of the physical location of the object. Additionally or alternatively, a returning action may be identified based on the difference between the first and second visual input, The returning action may comprise removing the object from the shopping cart, returning the object to a location within the store, or the like. The returning action may be indicative of the subject decision not to purchase the object after being determined as item for sale.

[0176] Additionally or alternatively, the background of the first visual input may be visually similar to the background of the second visual. The difference between the first context and the second context may be a non-visual difference. In some exemplary embodiments, the non-visual difference between the first context and the second context may be determined based on a difference between a first value associated with the first visual input and a second value associated with the second visual input, wherein the first and the second values are determined based on a third input obtained from a second sensor, such as a positioning reading of the wearable device during performing the action, accelerometer readings, or the like.

[0177] As an example, the last identified physical location of the object and the next identified physical location of the object may be similar. According to a movement pattern of the hand which may be determined based on accelerometry data, the action may be determined to be taking the object off the original location and return it back to its original location. Additional analysis of the foreground of the visual input may be performed in order to identify additional attributes of the action, such as re-arranging the object in its original location, or the like.

[0178] Referring now to Figure 5 showing a block diagram of a system, in accordance with some exemplary embodiments of the disclosed subject matter.

[0179] In some exemplary embodiments, a System 500 may be utilized to manage a self-service shopping of a User 505 in a store, or online-shopping fulfillment for customers by User 505, or the like. Additionally or alternatively, similar applications of System 500 may be utilized for other facilities to perform monitoring of hand actions of users, such as in health-care systems to monitor action of health-care staff members, in airplanes to monitor actions of pilots, in augmented reality video games to monitor actions of players, or the like.

[0180] In some exemplary embodiments, System 500 may comprise a plurality of Wearable Devices 510 each of which is being worn on or held by a user such as User 505. Each Wearable Device 510 may be configured to be worn by User 505 in a manner enabling monitoring of hand activity of User 505. Wearable Device 510 may be configured to be utilized to identify items grabbed by the hand of User 505 and moved to or from a one or more physical shopping totes of User 505. In some exemplary embodiments, Wearable Devices 510 may be worn on the hand of User 505, in a manner enabling capturing interior portion thereof, such as on a wrist of User 505, on fingers of User 505, on a hand palm of User 505, or the like.

[0181] In some exemplary embodiments, Wearable Device 510 may comprise a Visual Sensor 512. Visual Sensor 512 may be located in a location and orientation enabling monitoring of hand activity of User 505. Visual Sensor 512 may be configured to continuously capture an interior portion of the hand of User 505. Wearable Device 510 may be configured to provide visual input captured by Visual Sensor 512 to be utilized to identify activity performed by the hand of User 505, such as an action performed by the hand, an object upon which the action is performed, or the like. Visual Sensor 512 may comprise a single lens, one or more lenses, or the like. Visual Sensor 512 may be configured to capture pictures, videos, signals, a combination thereof, or the like. In some exemplary embodiments,

[0182] In some exemplary embodiments, Wearable Device 510 may comprise a Communication Unit 514. Communication Unit 514 may be configured to connect Wearable Device 510 to a controller external thereto, such as to a mobile Device 520 of User 505, Store Unit 530, to Server 540, or the like. Wearable Device 510 may be automatically activated when connected to Store Unit 530, such as based on connecting to a Wi-Fi network in the store associated with Store Unit 530, using an activation interface associated with Store Unit 530, based on the location readings of Wearable Device 510 being conformed with location of Store Unit 530, or the like. Similarly, Wearable Device 510 may be de-activated when leaving the store, such as based on disconnecting from Store Unit 530, based on store Unit identifying that User 505 left the store, or the like.

[0183] In some exemplary embodiments, an internal storage (not shown) may be utilized to retain images and data obtained from Visual Sensor 512. In some exemplary embodiments, Visual Sensor 512 may capture data when motion-related sensors (not shown), such as accelerometer, gyroscope, or the like, indicate that Wearable Device 510 is in motion. When the device is resting, e.g., when placed and not worn by a user, the device may avoid capturing video to preserve power resources. In some exemplary embodiments, images may not be captured in predetermined areas, such as rest rooms, lockers, or other private areas. Additionally or alternatively, power and network connectivity resources may be spared by initially transmitting low-quality versions of the images to Server 540 (e.g., directly or indirectly, such as via Store Unit 530). In case an initial analysis at the backend is performed and requires additional information, high- quality data may be transmitted. The quality of data may differ, for example, in the frame rate of a video segment, in the resolution of the images, or the like.

[0184] In some exemplary embodiments, Wearable Device 510 may comprise an Input/Output (VO) Module 516 configured to obtain input and provide output from Wearable Device 510 to other connected devices, such as providing visual input captured by Visual Sensor 512, readings of other sensors, or the like. [0185] In some exemplary embodiments, Wearable Device 510 may be associated with an application of a computing Device 520 of User 505, such as a mobile app, or the like. The mobile app may be a standalone native app, a feature embedded in or hosted by third party app(s), or the like. User 505 may receive data associated with the shopping session to Device 520, provide feedback, or the like. The data may be provided in real time or post actions. In some exemplary embodiments, the data may be displayed on a screen of Device 520, using the designated application or the like. As an example, Device 520 may be utilized to display a Virtual Cart Display 522 for User 505, upon initiating a selfshopping session, or an online- shopping manual fulfilment process, indicating the items shopped thereby. Additionally or alternatively, Device 520 may be utilized to display a Shopping List 524 for User 505. Additionally or alternatively, Device 520 may be attached to or embedded with Wearable Device 510, such as in a smart watch, a smart wristband with a touch screen, or the like.

[0186] In some exemplary embodiments, System 500 may comprise a Server 540. Server 540 may be configured to support the monitoring and identification of hand actions of users in the store, such as User 505, to perform respective responsive actions, to issue output to User 505 or to Store Unit 530, or the like.

[0187] In some exemplary embodiments, Server 540 may comprise a Foreground Identification Module 545. Foreground Identification Module 545 may be configured to identify a foreground of a given visual input, distinguish between a foreground and a background of the given visual input, or the like. In some exemplary embodiments, Foreground Identification Module 545 may be configured to identify pixels of the visual input that may be associated with a foreground or a background of the given visual input, such as using vision analysis techniques, machine learning techniques, or the like.

[0188] In some exemplary embodiments, Server 540 may comprise an Object Identification Module 550. Object Identification Module 550 may be configured to identify an object in a foreground of a given visual input. In some exemplary embodiments, Object Identification Module 550 may be configured to identify any physical object, such as all physical objects in the foreground of the visual input. Additionally or alternatively, Object Identification Module 550 may be configured to identify a certain physical object or a portion thereof, such as identifying a predetermined object being analyzed, identifying an objects identified in previous or other visual input, or the like. Object Identification Module 550 may be configured to identify the object based on visual characteristics, such as identifying a predetermined shape or color, identifying a barcode or other unique identifier of the object, or the like.

[0189] In some exemplary embodiments, Server 540 may comprise a Catalog Database 580 retaining visual representations of items in the store. Object Identification Module 550 may be configured to recognize the object which upon the hand of User 505 performs the action based on Catalog Database 580. Catalog Database 580 may be retained by System 500. Object Identification Module 550 may be configured to compare and match objects identified in the input with objects of items stored in Catalog Database 580. Prior to activating the solution, all items which may be sold in the store may be pictured from different angels, categorized and stored in Catalog Database 580. When Server 540 recognizes that the hand(s) of User 505 made an action or got close to an item, Object Identification Module 550 may be configured to match the picture, video or signal of the item may be matched with Catalog Database 580, and identified thereof (or not identified, if such item is not listed in the database or matching has not succeeded). Additionally or alternatively, Catalog Database 580 may comprise a plurality of partial databases for each store. The partial database may comprise items that are known to be in the store in a certain location. In order to speed the identification of an item, instead of searching the entire database each time, only the items, which the service is aware of being located in the store at the certain location, or located next to the location of Wearable Device 510, may be searched. In case that Object Identification Module 550 does not find the item in that partial database, Object Identification Module 550 may search the whole Catalog Database 580 for that item. Object Identification Module 550 may be configured to know what items are located in certain stores based on an inventory list obtained from the User 505, Store Unit 530, or the like. Additionally or alternatively, Object Identification Module 550 may be configured to know what items are located in a certain store based on information obtained from wearable device used by the retailer’s worker upon arranging the store. Each item that is located in the store may be identified and listed in the partial database of the store. Additionally or alternatively, Object Identification Module 550 may be configured to identify parameters of the object, such as type, category, name, shape, size, or the like. Such parameters may be identified based on data retained in Catalog Database 580, or other databases.

[0190] In some exemplary embodiments, Server 540 may be configured to apply machine learning techniques, classification techniques, image processing techniques, Al techniques, or the like, in order to identify the object and the action performed thereon, to identify context of the visual input, to learn behavior of User 505, shopping habits thereof, or the like. Server 540 may be configured to collect and store information related to new or existing items and objects, such as pictures, videos, signals, classifications, or the like, in Catalog Database 580. Server may be configured to improve the recognition and identification of items and objects of System 500, context identification, actions made by User 505, or the like.

[0191] In some exemplary embodiments, Server 540 may be configured to obtain a first and a second visual inputs from Visual Sensor 512. The first and second visual inputs may be connected, such as capturing the same environment in subsequent times, capturing related environments, related to the same object, or the like. In some exemplary embodiments, Server 540 may be configured to analyze the first and the second input in order to identify an action performed by the hands of User 505 and an object which upon the action is performed. Object Identification Module 550 may be configured to identify items or objects withing the visual input. Action Identification Module 560 may be configured to identify actions performed on the identified objects by the hand of User 505, or actions refrained from being done thereon. Additionally or alternatively, the action may be associated with modifying content of a physical shopping tote of User 505. Action Identification Module 560 may be configured to recognize when the hand of User 505 is getting close to an item, picking an item, holding an item (as an example, while the object stays constant at the hand), moving an item (as an example, background picture changed), releasing an item, or the like.

[0192] In some exemplary embodiments, Object Identification Module 550 may be configured to identify the object or portions thereof in a foreground of the first visual input. Action Identification Module 560 may be configured to determine that User 505 is holding the object in the first visual input, based on Object Identification Module 550 identifying the object in the foreground of the first visual input. In some exemplary embodiments, Action Identification Module 560 may be configured to identify a handbased action of User 505 with respect to the object based on a difference between the first visual input and the second visual input. In some exemplary embodiments, Action Identification Module 560 may be configured to identify the hand-based action of the subject with respect to the object based on Object Identification Module 550 identifying that the object is not in a foreground of the second visual input. [0193] In some exemplary embodiments, Server 540 may comprise a Context Determination Module 555. Context Determination Module 555 may be configured to determine a context of the object based on a background thereof in the visual input. As an example, Context Determination Module 555 may be configured to determine a first context of the object based on a background of the first visual input and a second context of the object based on a background of the second visual input. Action Identification Module 560 may be configured to identify the hand-based action of the subject with respect to the object based on a difference between the first context and the second context. As an example, Action Identification Module 560 may be configured determine that User 505 is holding the object in the second visual input based on an identification of the object in a foreground of the second visual input, after being identified in the foreground of the first visual input. Action Identification Module 560 may be configured to identify the hand-based action of the subject with respect to the object by determining that a physical location of the object has changed based on the difference between the first context and the second context. Action Identification Module 560 may be configured to determining the hand-based action based on the change of the physical location of the object. As another example, Action Identification Module 560 may be configured to determine that User 505 is picking up the object based on the first context of the object being associated with a location of the object in the store, and the second context being related to a shopping tote or cart.

[0194] Additionally or alternatively, Context Determination Module 555 may be configured to determining the context of the visual input based on additional sensory data obtained from other sensors located on Wearable Device 510 or Device 520, such as an accelerometer, a gyroscope, or the like. It may be noted that in some cases the background of the first visual input may be visually similar to the background of the second visual, and the difference between the first context and the second context may be a non- visual difference, such as a difference in accelerometry, or the like. Additionally or alternatively, in some cases the background of the visual input may be blocked by the object. The context of the visual input may be determined based on the background of previous or next visual input, based on other types of sensory data, based on a change of orientation of the object, or the like.

[0195] Additionally or alternatively, Action Identification Module 560 may be configured to identify the action being performed on the object is pick-up action, based on Object Identification Module 550 identifying that the object appears in a background of the second visual input, and then appearing in a foreground of the second visual input. Additionally or alternatively, Action Identification Module 560 may be configured to identify the action being performed on the object is placement action, based on Object Identification Module 550 identifying that the object appearing a foreground of the first visual input and then appearing in a background of the second visual input.

[0196] Additionally or alternatively, Object Identification Module 550 may be configured to obtain visual input of the content of the physical shopping tote, such as from Wearable Device 510, or other visual sensors of the store associated with Store Unit 530, sensors of User Device 520, or the like. Control Module 570 may be configured to determine a discrepancy between content of Virtual Cart 522 and the content of the physical shopping tote, such as based on identifying the items in the physical shopping tote and comparing the identified items to the items listed in in Virtual Cart 522. Control Module 570 may be configured to perform a responsive action in response to the determined discrepancy, such as by marking Virtual Cart 522 as invalidated, updating Virtual Cart 522 based on the visual input of the content of the physical shopping tote, or the like.

[0197] In some exemplary embodiments, Object Identification Module 550 may be configured to determine that the object appears in the foreground of the first visual input based a sequence of visual inputs that comprise the first visual input, by identifying a pattern of appearance of the object in the sequence of visual inputs, a movement pattern of the hand of the subject with respect to the object in the sequence of visual inputs, or the like

[0198] Additionally or alternatively, the visual inputs obtained from Visual Sensors 512 may comprise a first visual input, a second visual input, a third visual input and a fourth visual input. The first visual input may precede the second visual input by no more than a first predetermined threshold and the third visual input may precede the fourth visual input by no more than a second predetermined threshold. The first predetermined threshold and the second predetermined threshold may be identical. The first predetermined threshold or the second predetermined threshold may be selected from a group consisting of 1 second, 500 milliseconds, 200 milliseconds, 100 milliseconds, 50 milliseconds, 30 frames, 10 frames, 5 frames, and 1 frame. [0199] Action Identification Module 560 may be configured to automatically identify a pick action by the hand with respect to an object based on the first visual input and the second visual input, based on a determination that in the first visual input the hand not touching the object and based on a determination that in the second visual input the hand touches the object. Additionally or alternatively, automatically identify a release action by the hand with respect to the object based on the third visual input and the fourth visual input, based on a determination that in the third visual input the hand touches the object and based on a determination that in the fourth visual input the hand does not the object.

[0200] In some exemplary embodiments, Store Unit 530 may comprise a mapping module that may be configured to update digital mapping of objects to locations based on the automatic identification of the pick action and release action by Action Identification Module 560, based on the context of the objects determined by Context Identification Module 555, or the like. As an example, the mapping module may be configured to update a location of a digital representation of the object from a first location to a second location identified based on the identified action. The first location may be identified based on a first context of the object based on a background of the first visual input or the second visual input; and the second location may be identified based on a second context of the object identified based on a background of the third visual input or the fourth visual input.

[0201] In some exemplary embodiments, activation and de-activation of Wearable Device 510 may be performed automatically by Control Module 570 of Server 540. Wearable Device 510 may be devoid of a de-activation interface for User 505. The activation of Wearable Device 510 may be performed in response to identifying a checkin activity associated with User 505, such as a connection from Wearable Device 510 to Store Unit 530, an indication from Control Module 570 that Wearable Device 510 is collected by User 505, based on a pairing between Wearable Device 520 and Store Unit 530, or the like. Similarly, the deactivation of Wearable Device 510 may be performed in response to determining a check-out activity associated with User 505. Additionally or alternatively, Control Module 570 may be configured to associate Wearable Device 510 being worn by a certain picker and the respective customer orders being collected by the picker, obtaining the associated Customer Item List 522, updating the associated Customer Virtual Cart 522, or the like. [0202] Additionally or alternatively, Control Module 570 may be configured to determine a mapping of geo-spatial locations of items in the store. Control Module 570 may be configured to identify a placement location of each object moved by User 505 or any other user, such as a worker in the store, and update the mapping to indicate a location of the object based on the placement location.

[0203] In some exemplary embodiments, Wearable Device 510 may be configured to be utilized for manual fulfillment of a shopping order of a customer by User 505. User 505 may be a picker tasked with picking items to fulfill the shopping order of the customer. Control Module 570 may be configured to identify that the object identified by Object Identification Module 550 is associated with the order of the customer. In some exemplary embodiments, the shopping order may comprise a List 524 of items selected by the customer and transmitted to Device 520 of User 505. Action Identification Module 560 may be configured to identify a picking captured by Wearable Device 510, such as picking up an object and placing the object in a tote associated with the shopping order of the customer. Control Module 570 may be configured to identify a corresponding item to the item in the shopping order (e.g. in List 524) and mark the corresponding item as fulfilled. In response to a determination that the shopping order is fulfilled, Control Module 570 may be configured to perform a responsive action, such as invoking a payment module (not shown) to enable a transaction from the customer to Store Unit 530, based on the fulfilled shopping order, or a portion thereof determined as fulfilled. In response to Action Identification Module 560 identifying that that User 505 (the picker) picked up the object and placed the object in a tote associated with the order of the customer, Control Module 570 may be configured to identify performing a fulfillment- related action. Additionally or alternatively, Control Module 570 may be configured to identify a mismatch between the object and Customer Items List 524 associated with the customer. In response to Action Identification Module 560 identifying that User 505 (the picker) picked up the object and placed the object in a tote associated with the order of the customer, Control Module 570 may be configured to issue an alert to the picker indicating the mismatch.

[0204] In some exemplary embodiments, Control Module 570 may be configured to determine a responsive action based on the action or the object. Control Module 570 may be configured to update one or more Virtual Carts 522 in response to the identification of the action by the user, such as adding and item, removing an item, or the like. Virtual Cart 522 may indicate a list of items shopped by User 505 for a certain customer or according to a certain Customer Items List 524, or the like. Virtual Cart 522 may be automatically updated based on items moved to and from the physical shopping tote of User 505, such as by adding items to Virtual Cart 522 based on items picked up and put into the physical shopping tote of the certain customer order being picked by User 505 and removing items from Virtual Cart 522 based on items removed from the physical shopping tote of the certain customer order being picked by User 505.

[0205] In some exemplary embodiments, Control Module 570 may be configured to issue an output to User 505. The output may be issued via I/O Module 516, to Device 520 of User 505, such as by displaying the content of Virtual Cart 522 to User 505 using Device 520 (Customer Virtual Cart 522), or the like, such as issuing an audio alert using a speaker on Wearable Device 510 or Device 520, using a LED light bulb on Wearable Device 510 or Device 520, or any other visual output, to provide an indication of an addition of an item to or removal of an item from Virtual Cart 522. Additionally or alternatively, Control Module 570 may be configured to issue an output to the customer whose order being picked by User 505, such as providing updated prices, suggesting alternative items, or the like.

[0206] The present disclosed subject matter may be a system, a method, and/or a computer program product. The computer program product may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present disclosed subject matter.

[0207] The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non- exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.

[0208] Computer readable program instructions described herein can be downloaded to respective computing/processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and/or a wireless network. The network may comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers. A network adapter card or network interface in each computing/processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing/processing device.

[0209] Computer readable program instructions for carrying out operations of the present disclosed subject matter may be assembler instructions, instruction-set- architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) may execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present disclosed subject matter.

[0210] Aspects of the present disclosed subject matter are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the disclosed subject matter. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer readable program instructions.

[0211] These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer, a programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein comprises an article of manufacture including instructions which implement aspects of the function/act specified in the flowchart and/or block diagram block or blocks.

[0212] The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process, such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions/acts specified in the flowchart and/or block diagram block or blocks.

[0213] The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosed subject matter. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts or carry out combinations of special purpose hardware and computer instructions.

[0214] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosed subject matter. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

[0215] The corresponding structures, materials, acts, and equivalents of all means or step plus function elements in the claims below are intended to include any structure, material, or act for performing the function in combination with other claimed elements as specifically claimed. The description of the present disclosed subject matter has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the disclosed subject matter in the form disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the disclosed subject matter. The embodiment was chosen and described in order to best explain the principles of the disclosed subject matter and the practical application, and to enable others of ordinary skill in the art to understand the disclosed subject matter for various embodiments with various modifications as are suited to the particular use contemplated.

Claims

What is claimed is:

1. A method comprising: obtaining a first visual input and a second visual input from a sensor located on a wearable device, the wearable device is worn by a subject on a hand of the subject, the sensor is configured to be placed in a location and orientation enabling monitoring of hand activity of the subject; determining that the subject is holding an object in the first visual input, wherein said determining that the subject is holding the object is based on an identification of the object in a foreground of the first visual input; and automatically identifying a hand-based action of the subject with respect to the object based on a difference between the first visual input and the second visual input.

2. The method of Claim 1, wherein said automatically identifying the hand-based action of the subject with respect to the object comprises: identifying that the object is not in a foreground of the second visual input.

3. The method of Claim 1 comprises determining that the subject is holding the object in the second visual input based on an identification of the object in a foreground of the second visual input.

4. The method of Claim 1, wherein said automatically identifying the hand-based action of the subject with respect to the object comprises: determining a first context of the object based on a background of the first visual input; determining a second context of the object based on a background of the second visual input; and identifying the hand-based action of the subject with respect to the object based on a difference between the first context and the second context.

5. The method of Claim 4, wherein said automatically identifying the hand-based action of the subject with respect to the object further comprises:

65 determining that a physical location of the object has changed based on the difference between the first context and the second context; and determining the hand-based action based on the change of the physical location of the object.

6. The method of Claim 4, wherein said determining the second context is further performed based on a third input obtained from a second sensor located on the wearable device.

7. The method of Claim 4, wherein the background of the first visual input is visually similar to the background of the second visual, wherein the difference between the first context and the second context is a non-visual difference.

8. The method of Claim 7, wherein the non-visual difference between the first context and the second context is determined based on a difference between a first value associated with the first visual input and a second value associated with the second visual input, wherein the first and the second values are determined based on a third input obtained from a second sensor.

9. The method of Claim 1, wherein the first visual input comprises at least a first portion of the object; wherein the second visual input comprises at least a second portion of the object; wherein said automatically identifying the hand-based action of the subject with respect to the object comprises: identifying that the second portion of the object blocks a background of the second visual input.

10. The method of Claim 1, wherein the second visual input is obtained prior to the first visual input, wherein the second visual input comprises the object; determining that the object appears in a background of the second visual input; and wherein the hand-based action of the subject with respect to the object is a pick-up action of the object by the subject.

66

11. The method of Claim 1 , wherein the second visual input succeeds the first visual input, wherein the second visual input comprises the object; determining that the object appears in a background of the second visual input; and wherein the hand-based action of the subject with respect to the object is a placement action of the object by the subject.

12. The method of Claim 1, wherein said determining that the object appears in the foreground of the first visual input is performed based a sequence of visual inputs that comprise the first visual input, wherein said determining that the object appears in the foreground comprises: identifying a pattern of appearance of the object in the sequence of visual inputs.

13. The method of Claim 1, wherein said determining that the object appears in the foreground of the first visual input is performed based a sequence of visual inputs that comprise the first visual input, wherein said determining that the object appears in the foreground comprises: identifying a movement pattern of the hand of the subject with respect to the object in the sequence of visual inputs.

14. The method of Claim 1, wherein the subject is a picker tasked with picking items to fulfill an order of a customer.

15. The method of Claim 14, further comprises: identifying that the object is associated with the order of the customer; wherein said automatically identifying the hand-based action of the subject with respect to the object comprises: identifying that the picker picked up the object and placed the object in a tote associated with the order of the customer; and in response to said automatically identifying: performing a fulfillment- related action.

16. The method of Claim 14, further comprises: identifying a mismatch between the object and a list of items in the order of the customer;

67 wherein said automatically identifying the hand-based action of the subject with respect to the object comprises: identifying that the picker picked up the object and placed the object in a tote associated with the order of the customer; and in response to said automatically identifying: issuing an alert to the picker indicating the mismatch. The method of Claim 1, wherein a view of the sensor is blocked, at least in part, by the hand. A method comprising: obtaining visual inputs from a sensor located on a wearable device, the wearable device is worn by a subject on a hand of the subject, the sensor is configured to be placed in a location and orientation enabling monitoring of hand activity of the subject, a view of the sensor is blocked, at least in part, by the hand, the visual inputs comprise a first visual input, a second visual input, a third visual input and a fourth visual input, the first visual input precedes the second visual input, the third visual input precedes the fourth visual input; automatically identifying a pick action by the hand with respect to an object based on the first visual input and the second visual input, wherein said automatically identifying the pick action is based on a determination that in the first visual input the hand not touching the object and based on a determination that in the second visual input the hand touches the object; and automatically identifying a release action by the hand with respect to the object based on the third visual input and the fourth visual input, wherein said automatically identifying the release action is based on a determination that in the third visual input the hand touches the object and based on a determination that in the fourth visual input the hand does not the object. The method of Claim 18 further comprises: based on said automatically identifying the pick action and based on said automatically identifying the release action, updating a digital mapping of objects to locations, wherein said updating updates a location of a digital representation of the object from a first location to a second location.

68

20. The method of Claim 18 further comprises: determining a first context of the object based on a background of the first visual input or the second visual input, the first context is associated with the first location; and determining a second context of the object based on a background of the third visual input or the fourth visual input, the second context is associated with the second location.

21. The method of Claim 20, wherein the second location is determined based on the second context.

22. The method of Claim 18, wherein the first visual input precedes the second visual input by no more than a first predetermined threshold, the third visual input precedes the fourth visual input by no more than a second predetermined threshold.

23. The method of Claim 22, wherein the first predetermined threshold and the second predetermined threshold are identical.

24. The method of Claim 22, wherein the first predetermined threshold or the second predetermined threshold are selected from a group consisting of: no more than 1 second; no more than 500 milliseconds; no more than 200 milliseconds; no more than 100 milliseconds; no more than 50 milliseconds; no more than 30 frames; no more than 10 frames; no more than 5 frames; and no more than a single frame.

25. A computerized apparatus having a processor, the processor being adapted to perform the steps of: obtaining a first visual input and a second visual input from a sensor located on a wearable device, the wearable device is worn by a subject on a hand of the

69 subject, the sensor is configured to be placed in a location and orientation enabling monitoring of hand activity of the subject; determining that the subject is holding an object in the first visual input, wherein said determining that the subject is holding the object is based on an identification of the object in a foreground of the first visual input; and automatically identifying a hand-based action of the subject with respect to the object based on a difference between the first visual input and the second visual input. A computer program product comprising a non-transitory computer readable storage medium retaining program instructions, which program instructions when read by a processor, cause the processor to perform a method comprising: obtaining a first visual input and a second visual input from a sensor located on a wearable device, the wearable device is worn by a subject on a hand of the subject, the sensor is configured to be placed in a location and orientation enabling monitoring of hand activity of the subject; determining that the subject is holding an object in the first visual input, wherein said determining that the subject is holding the object is based on an identification of the object in a foreground of the first visual input; and automatically identifying a hand-based action of the subject with respect to the object based on a difference between the first visual input and the second visual input.

70