KR20220082011A - Systems and methods for personalized navigation within a business enterprise - Google Patents

Abstract

Systems and methods for tracking the movement of individuals through a building include receiving RF signals from RF transmitting mobile devices carried by people near an entrance by one or more RF nodes disposed near an entrance to a building. receiving, capturing images of people while they are near the entrance, and determining the identity and relative distance of each RF transmitting mobile device from each RF node based on information associated with the RF signals received by the RF node determine, detect humans in the image, determine a relative depth of each human in the image, and determine each human based on the relative distance of each RF transmitting mobile device from each RF node and the relative depth of each human in the image. Assigns the identity of the RF transmitting mobile device to one of the humans detected in the image, thereby optically identifying each individual to be tracked as the individual moves throughout the building.

Description

Systems and methods for personalized navigation within a business enterprise

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation-in-part of pending U.S. Patent Application Serial No. 16/163,708, filed October 18, 2018, entitled "System and Method of Personalized Navigation Inside a Business Enterprise," and this application was filed in 2017 Claims the benefit of, and priority to, a portion of the continuation of U.S. Patent Application Serial No. 15/839,298, filed December 12, , entitled "System and Method of Personalized Navigation Inside a Business Enterprise," and this application is entitled to 35 U.S.C. Claims the benefit of and priority to U.S. Provisional Application No. 62/432,876, filed December 12, 2016, and entitled "System and Method of Personalized Navigation Inside a Business Enterprise," under § 119(e); All applications and provisional applications are incorporated herein by reference for all purposes.

technical field

The present invention relates generally to systems and methods for providing personalized navigation for people while inside a certain business enterprise.

A common complaint among shoppers is that they are often dissatisfied by not knowing where an item is located within the store. They roam inefficiently through aisles, looking for items on their shopping lists, often tracing back the way they came, and taking long routes in their quest for desired items.

According to one embodiment, the present invention relates to a system for tracking the location of an individual within a building. The system includes at least one radiofrequency (RF) node disposed near an entrance to a building. At least one RF node has an RF receiver for receiving RF signals from RF transmitting devices near an entrance to a building. At least one optical device is disposed near an entrance to the building, and the at least one optical device captures an image of the plurality of people while the plurality of people are near the entrance to the building. The controller communicates with the at least one RF node to obtain information therefrom associated with RF signals received by an RF receiver of the at least one RF node, and communicates with the at least one optical device to obtain a captured image therefrom. The controller determines the identity of each RF transmitting device and the angular position of the RF transmitting device with respect to each node of the at least one RF node and the angular position of the RF transmitting device with respect to each RF node of the at least one RF node. , determine based on information associated with RF signals obtained by a controller from the RF node of at least one RF node. The controller determines the identity of each RF transmitting device to detect the plurality of humans and the orientation marker in the image obtained by the controller from the at least one optical device, and the position of the orientation marker in the image for each human detected in the image. and based on the determined angular position of the RF transmitting device relative to each RF node of the at least one RF node, to one human of the plurality of humans detected in the image. Thereby, each individual to be located is optically identified as said individual moves throughout the building.

In one exemplary embodiment, the at least one RF node is configured to determine a relative signal strength indicator (RSSI) value for RF signals received from each RF transmitting device. The information obtained by the controller from the at least one RF node includes RSSI values. The controller is configured to estimate a distance of each RF transmitting device from each RF node based on RSSI values for RF signals received by the RF node from the RF transmitting device, and an identity of each RF transmitting device. and use the estimated distance of each RF transmitting device from each RF node when assigning to one of the plurality of humans detected in the image.

In another exemplary embodiment, the controller is configured to detect in the image a mobile phone carried by a human of a plurality of humans detected in the image, and a mobile phone in which a human of a plurality of humans detected in the image was detected. and assign an identity of each RF transmitting device to one human of the plurality of humans detected in the captured image based on the possessing the .

In another exemplary embodiment, the controller is configured to: for each RF node arrange the RF transmitting mobile devices in an angular order based on the angular positions of the RF transmitting mobile devices from the RF node: to arrange the captured humans in the image in angular order based on the relative positions of the humans in and compare the angular order of the humans detected in the image with the angular order of the RF transmitting mobile devices.

The controller may be further configured to arrange the angular order of the plurality of humans in the captured image for the at least one RF node before the angular order of humans detected in the image is compared with the angular order of the RF transmitting mobile devices. .

In another exemplary embodiment, the system further comprises a plurality of optical devices disposed throughout the building, and wherein the controller selects each identified individual throughout the building, over time at least one of the plurality of optical devices. and optically track based on detecting the identified individual in images received from the portion.

In another exemplary embodiment, the information conveyed by RF signals transmitted by a given RF transmitting mobile device includes a shopping list. The controller is further configured to determine a route through the building based on the items on the shopping list and to transmit the route to the given RF transmitting mobile device for display on a screen of the given RF transmitting mobile device.

As another example, the system further includes a plurality of optical devices disposed throughout the building, and wherein the controller is configured to select each identified individual throughout the building as received from at least some of the plurality of optical devices over time. and optically track based on detecting the identified individual in images.

In other examples, the information conveyed by RF signals transmitted by a given RF transmitting mobile device includes a shopping list. The controller is further configured to determine a route through the building based on the items on the shopping list and to transmit the route to the given RF transmitting mobile device for display on a screen of the given RF transmitting mobile device. At least one RF node may include two or more RF nodes.

In other examples, the controller is configured to: calculate a position for each RF transmitting mobile device based on angular positions of the RF transmitting mobile devices received from the at least one RF node: to arrange the RF transmitting mobile devices in depth order based on the calculated positions of and assign the identity of each RF transmitting mobile device to one of the plurality of humans detected in the captured image by comparing the depth order of the detected humans to the depth order of the RF transmitting mobile devices.

According to another embodiment, the present invention relates to a method for tracking the location of an individual within a building. The method includes receiving, by at least one radio frequency (RF) node disposed near an entrance to a building, RF signals from RF transmitting mobile devices carried by a plurality of people near an entrance to the building. include While the plurality of people are near the entrance to the building, images of the plurality of people are captured. An identity of each RF transmitting mobile device and an angular position of each RF transmitting mobile device from each RF node are determined based on information associated with the RF signals received by the at least one RF node. A plurality of humans and orientation markers are detected in the captured image. The identity of each RF transmitting mobile device was captured based on the position of the orientation marker in the image for each human detected in the image and based on the determined angular position of each RF transmitting mobile device for each RF node. Assigned to one human of a plurality of humans detected in the image, thereby optically identifying each individual to be located as the individual moves throughout the building.

In one exemplary embodiment, a relative signal strength indicator (RSSI) value for RF signals received from each RF transmitting device is determined, and the estimated distance of each RF transmitting device from each RF node is Each RF is calculated based on RSSI values for RF signals received by the RF node from an RF transmitting device and when assigning the identity of each RF transmitting device to one of a plurality of humans detected in the image. The estimated distance of each RF transmitting device from the node is used.

In one exemplary embodiment, a mobile phone carried by one of the plurality of humans is detected in the image, and an identity of each RF transmitting device is assigned to one of the plurality of humans detected in the captured image. The step is based on the presence of the detected mobile phone of the human detected in the image.

In another exemplary embodiment, the RF transmitting mobile devices are arranged in an angular order based on the angular positions of the RF transmitting mobile devices with respect to the at least one RF node, and in relative positions of a plurality of humans detected in the captured image. wherein the humans captured in the image are arranged in angular order based on the angular order of the humans detected in the image, and assigning the identity of each RF transmitting mobile device to one of the plurality of humans detected in the captured image includes: and comparing the angular order of the RF transmitting mobile devices.

The angular order of humans in the captured image may be arranged for the at least one RF node before the angular order of humans detected in the captured image is compared with the angular order of the RF transmitting mobile devices.

In another exemplary embodiment, the location of each identified individual passing through a building is optically tracked by comparing successive images capturing the identified individual.

In another exemplary embodiment, information associated with RF signals transmitted by a given RF transmitting mobile device includes a shopping list. A route through the building may be determined based on items on the shopping list, and the route may be transmitted to a given RF transmitting mobile device for display on a screen of the given RF transmitting mobile device.

In another exemplary embodiment, wireless signals conveying the current location of a given identified individual within a building may be transmitted from at least one RF node to an RF transmitting mobile device carried by the given identified individual.

In another example embodiment, a position for each RF transmitting mobile device is calculated based on angular positions of the RF transmitting mobile devices received from the at least one RF node. The RF transmitting mobile devices are arranged in depth order based on the calculated positions of the RF transmitting mobile devices relative to the at least one RF node. Based on the relative positions of the humans detected in the captured image, the humans detected in the captured image are arranged in depth order. Assigning the identity of each RF transmitting mobile device to one of the plurality of humans detected in the captured image includes comparing a depth order of the humans detected in the image to a depth order of the RF transmitting mobile devices. .

According to another embodiment, the present invention relates to a system for tracking the location of an individual within a building. The system includes at least one radio frequency (RF) node disposed near an entrance to a building. The at least one RF node has an RF receiver channel for receiving RF signals from a plurality of RF transmitting devices near an entrance to the building. The system further includes at least one camera disposed near the entrance to the building. The at least one camera captures images of the plurality of people while they are near the entrance to the building. A controller of the system communicates with the at least one RF node to obtain information therefrom associated with RF signals received by an RF receiver channel of the at least one RF node, and communicates with the at least one optical device to obtain an image captured therefrom. to acquire The controller is configured to associate the identity of each RF transmitting device and the position of the RF transmitting device with respect to each RF node of the at least one RF node with the RF signals obtained by the controller from the RF node of the at least one RF node. to determine based on the information, to detect a plurality of humans in the image obtained by the controller from the at least one camera, to detect an orientation marker in the image, and to determine the identity of each RF transmitting device from each detected in the image. assigning to one human of a plurality of humans detected in the image based on the position of the orientation marker in the image for the human and based on the determined angular position of the RF transmitting device relative to each RF node of the at least one RF node is configured to

Still other aspects, embodiments, and advantages of these exemplary aspects and embodiments are discussed in detail below. Embodiments disclosed herein may be combined with other embodiments in any manner consistent with at least one of the principles disclosed herein, and may include “an example”, “an embodiment”, “some embodiments References to “, “alternative embodiment,” “various embodiments,” “one embodiment,” etc. are not necessarily mutually exclusive, and that a particular feature, structure, or characteristic described may depend on at least one embodiment. It is intended to indicate that it may be included. The appearances of such terms herein are not necessarily all referring to the same embodiment.

Various aspects of at least one embodiment are discussed below with reference to the accompanying drawings, which are not intended to be drawn to scale. The drawings are included to provide illustration and a further understanding of the various aspects and embodiments, and are incorporated in and constitute a part of this specification, but are not intended as definitions of the limitations of the invention. In the drawings, each identical or nearly identical component shown in the various drawings is represented by a same number. For purposes of clarity, not all components may be labeled in all figures. In the drawings,
1 is a block diagram of an example of a personalized navigation system in accordance with certain aspects of the present invention;
2 is an exemplary diagram of a top view of a business enterprise configured with a personalized navigation system, in accordance with certain aspects of the present invention;
3 is an exemplary display of a top view of a business enterprise as may appear on the screen of a mobile device carried by a shopper, in accordance with certain aspects of the present invention.
4 is a diagram illustrating an example of a system including a shelf camera and an aisle camera to track a shopper to the front of a retail shelf in a business enterprise, in accordance with aspects of the present invention.
5 is a diagram illustrating an example of a personalized navigation system configured to discriminate multiple people disposed at the entrance of a business enterprise.
6 is a flow diagram of one embodiment of a process for differentiating multiple people located at the entrance of a business enterprise.
7 is a flow diagram of another embodiment of a process for differentiating multiple people located at the entrance of a business enterprise.

Personalized navigation systems in accordance with certain aspects and embodiments provide a combination of radio frequency (RF) technology and optical imaging technology for identifying and tracking people in business enterprises, and providing people with personalized information and personalization within a business enterprise. Use software to provide a customized navigation experience. As discussed in more detail below, a person may, for example, provide information such as a shopping list or service request to a personalized navigation system, enabling the person to efficiently complete his purpose(s) in the business enterprise. In response to assisting may receive personalized navigation or other information. In this way, the human experience in the business enterprise may be improved.

According to one embodiment, the personalized navigation system uses RF technology to initially identify a shopper approaching an entrance to the business enterprise, and uses optical technology to initially identify the shopper after the shopper arrives and enters the business enterprise. Detect and track movement of To cooperate with the navigation system, the shopper carries a mobile device (eg, a smartphone or smart watch) with RF transmit and RF receive capabilities. In certain embodiments, the mobile device runs certain application software that transmits RF signals including the shopper's identity and the shopper's shopping list. A shopper may obtain the application software and download it to a mobile device from an “app store”. Many business enterprises currently have RF transmitters, RF receivers, and video cameras, and advantageously, the navigation systems described herein do not require any hardware modifications to such existing RF and video equipment. .

1 shows an example of a personalized navigation system 100 configured to operate in a business enterprise 110 having at least one RF node 120 . Any number of RF nodes designated by 120 -1 to 120-n (n being a positive integer) may be included. Each RF node 120-1 through 120-n includes a corresponding RF receiver antenna 130-1, 130-n (typically 130). Optionally, each RF node 120 may also have an RF transmit antenna (not shown). Each RF node 120 may be located at or near the entrance to the business enterprise 110 . Examples of business enterprise 110 include, but are not limited to, grocery stores, supermarkets, department stores, hardware stores, warehouses, and retail stores. In general, as described herein, if the mobile device 140 is running application software that provides personalized navigation of the business enterprise, the location of the RF node 120 near the entrance of the business enterprise 110 is Facilitates detection of an RF transmitting mobile device 140 associated with a person approaching an entrance. Mobile device 140 engages in communication with one or more of RF nodes 120 using a wireless communication technology such as Bluetooth®, Wi-Fi, or Near Field Communication (NFC). If the business enterprise 110 has multiple entrances, and at least one RF node 120 is deployed near each entrance, the personalized navigation system 100 determines which RF node 120 is the mobile device 140 . An entrance through which a person enters can be recognized based on whether the

During typical operation, a person with a mobile device 140 approaches an entrance to a business enterprise (ie, a building) 110 . The mobile device 140 runs the personalized navigation app and transmits RF signals. In certain examples, the RF signals carry an identifier associated with a person, whereby the operator of the business enterprise 110 knows the person. For example, the identifier may include a person's name, phone number, rewards program number associated with the business enterprise, or other identifying information. The RF signals may also carry a person's shopping list that identifies those items that the person wishes to find when visiting the business enterprise 110 . Typically, a person may prepare such a shopping list prior to visiting the business enterprise 110 ; The shopping list may be constructed or edited at any time before or after a person arrives at the business enterprise 110 .

When a person comes within range of the RF receiver antenna 130 , the mobile device 140 establishes communication with the RF node 120 . In particular, in certain examples, mobile device 140 may communicate the identifier and shopping list to RF node 120 . The mobile device may also or alternatively communicate other data to the RF node 120 , such as a set of instructions or other information, for example, for a technician or similar repair staff performing certain services in the business enterprise 110 . have. In certain examples, the RF node 120 forwards the identifier and shopping list or other data to a computer processing unit (also referred to as a controller) 150, which uses the identifier to communicate to the person associated with the identifier. You can access the database 160 in which information is stored. Such information may include, for example, records of prior visits to the business enterprise 110 by a person. Although computer processing unit 150 and database 160 are shown in FIG. 1 as within business enterprise 110, they are instead located in other locations, such as of computers and servers broadly referred to as the “cloud”. It may exist on a third-party network.

2 shows various aisles 210 - 1 to 210 - 7 (alternatively) arranged throughout the store to provide full coverage of the interior of the business enterprise 100 , preferably wherever visitors can walk. , shows a simplified example of a top view 200 of a business enterprise 100 with 210 and video cameras 220-1 through 220-n (typically 220). The business enterprise 100 may also include one or more RF transmitters 230 and a networking device 240 . Each video camera 220 , each RF node 120 , and each RF transmitter 230 communicates with the networking device 240 by a wireless or wired communication link (not shown). Networking device 240 communicates with computer processing unit 150 (shown in FIG. 1 as discussed above), which in one embodiment resides on cloud 250 . A wired or wireless communication link 260 connects the networking device 240 to the central processing unit 150 .

As discussed above, tracking of people within the business enterprise 110 uses optical technology; In particular, it may be accomplished by capturing and processing images from video cameras 220 located throughout the business enterprise 110 . According to one embodiment, during typical operation of a personalized navigation system, video cameras 220 continuously capture images within their field of view. At least one video camera 220 may be disposed proximate to each entrance of the business enterprise 110 to acquire images of people entering the business enterprise. In some embodiments, multiple video cameras 220 provide a full field of view, or at least a functionally sufficient field of view, of the area around the entrance so that images of all people entering the business enterprise 110 can be acquired. in a manner adjacent to each inlet. As discussed above, at the entrance to the business enterprise 110 a person (referred to as a tracked person) having a mobile device 140 configured to run application software to engage with the personalized navigation system 110 . Upon arrival, the RF node 120 at the entrance receives an identifier and, optionally, other information (eg, a shopping list) from the mobile device 140 . At the same time, the video camera(s) 220 proximate the entrance captures images of the area around the entrance, at least some of these images should include the person being tracked. As discussed above, in certain examples, the computer processing unit 150 determines the entrance through which the person entered the enterprise 110 based on the RF node 120 that detected the person and the known locations of each RF node. Know. Accordingly, the computer processing unit 150 knows which video camera or cameras 220 are in position to capture images of the individual. These video cameras 220 pass their captured images to a networking device 240 , which sends the captured images to a central processing unit 150 . Central processing unit 150 includes an image processor that performs image processing techniques adapted to detect a person in an image and to associate the detected person with a most recently obtained identifier and shopping list.

Techniques for processing images to identify a person in the images are known in the art, and any such image processing techniques may be implemented by central processing unit 150 . For example, if the smartphone is in the hand of an individual, which may indicate that the individual is engaged in the personalized navigation system 100 , the image processor may display a video camera 220 - 1 positioned at the relevant entrance. may be adapted to examine images captured by Alternatively or in combination, the image processor may be adapted to examine captured images of a person's head or both hands. Since the central processing unit 150 expects that the next person within the field of view of the video camera 220-1 located at the entrance will be the same as the person who communicated with the RF node 120-1 located at the entrance, the The detected person is associated with the received identifier and shopping list. Once a person has been identified in the image and associated with the received identifier, the personalized navigation system 100 tracks and guides the person as they move through the business enterprise 110 .

Tracking may be accomplished by collecting images from various video cameras 220 located between the aisles 210 and processing these images to follow the person being tracked. In certain examples, the central processing unit 150 tracks the movement of the tracked person as the person being tracked moves from one camera field of view to another, so that the location of the tracked person within the business enterprise 110 can be dynamically adjusted. Register and update In one example, video cameras 120 operate in parallel, with all or some of the video cameras simultaneously providing images to a central processing unit. The images may be merged into a map or layout of the business enterprise 110 , as shown in FIG. 2 . In some examples, two or more of the video cameras 220 may have at least partially overlapping fields of view, and in other examples, different video cameras 220 are used to monitor different areas of the business enterprise 110 . do. Video cameras 220 may capture images from different perspectives. The central processing unit may flatten the images by removing perspective distortion from each of the images, and merge the resulting corrected images into a map. Image stitching techniques may be used to merge images.

In certain examples, the image stitching process first performs image alignment using algorithms capable of discovering relationships between images with varying degrees of overlap. These algorithms are suitable for applications such as video stabilization, summarization, and creation of panoramic mosaics that can be used in images taken from cameras 220 . After alignment is complete, image stitching algorithms are alerted to potential issues such as blurring or ghosting caused by parallax and scene movements as well as changing image exposures within the business enterprise 110 . In doing so, it takes the estimates generated by such algorithms and blends the images in a seamless manner. Various image stitching algorithms and processes are known in the art, and any such image processing techniques may be implemented by central processing unit 150 .

Handoff may occur when the person being tracked moves from one viewpoint to another or is seen by one camera 220 and not by other cameras. These handoffs may be made using images running in parallel on the central processing unit 150 , where the position and movement of the person being tracked is determined by which camera 220 has the best view of the person being tracked. It is determined by the central processing unit to use.

In certain examples, video cameras 220 may include depth sensors. In such examples, the image stitching operation may be omitted, and each camera stream data is processed independently for change, person detection and recognition. The resulting “region of interest” is then transformed into individual point clouds (described further below) and transformed into a single common coordinate system. The translational and rotational transformations used in this process are based on the position and orientation of the video cameras (and their associated depth sensors) with respect to each other. In one example, one camera is selected as the main sensor and all other camera data is transformed into the main coordinate system to achieve the same end result as the image stitching procedure: integration of the actual position of the person being tracked between the sensors.

In some examples, central processing unit 150 may use known information about a floor plan of a business enterprise to help identify and track people based on images obtained from video cameras 220 . For example, the central processing unit may use known shapes and positions of shelves along the aisles 210 to provide reference points. At times, the person being tracked may be obstructed from the view of the camera by, for example, another person, equipment or shelf. The personalized navigation system 100 may be configured to store a predetermined position of the tracked person and to compare multiple image frames to reposition the tracked person after temporary occlusion. As discussed further below, the personalized navigation system 100 may be configured to provide a suggested route for a tracked person through the business enterprise 110 , so that the central processing unit can be used to reposition a person after temporary occlusion using the predicted future position of

According to certain embodiments, central processing unit 150 may execute an image processing process optionally supplemented with depth information to track a person as discussed above. A two-dimensional (2D) optical image capture device with a single aperture (ie, video camera 220 ) can capture 2D image information on a plane (film, CCD, etc.). Acquiring three-dimensional (3D) information typically requires the acquisition of additional data. The three-dimensional data may be obtained using multiple video cameras 220 or by combining one or more video cameras with one or more depth sensors. Video cameras 220 may utilize visible light, infrared light, or other optical wavelength ranges. Depth sensors may be based on infrared, laser or other wavelength emitters that transmit light to an object. Depth sensors typically determine the distance to an object from which light is reflected or backscattered. Alternatively, depth sensors may utilize acoustic signals to determine the distance. In one embodiment, depth sensing is integrated into the video cameras 220 .

Image frames are obtained from video cameras 220 . Video camera systems with depth sensing capability typically output video (eg, RGB, CYMG) and depth field information. The video may optionally be encoded in a well-known format such as MPEG. The optical and depth information are stitched together. Open libraries such as OpenCV or OpenNI (used to capture depth images) allow optical and depth information to be stitched together. Alternatively, a user of the personalized navigation system 100 may develop custom software for generating 3D information or object data generated by the optical images and depth sensors.

An initial calibration may be performed over multiple image frames to determine background information for both the 2D optical images and depth sensing. During calibration, any motion (eg, people) is extracted or ignored during background extraction, until stable background optics (RGB) and depth information can be stored, for example, in database 160 . Calibration may be performed periodically or may be initiated by the personalized navigation system 100 , for example, when errors are detected.

After calibration is complete, the generated spatial filter masks can be used to extract a “region of interest” for each video camera 220 . For example, for a video camera located near an entrance to a business enterprise 110 , the region of interest may correspond to the region between the background and the foreground (the region in which a person is expected to be), and thus a wall, door, or Anything other than any other infrastructure (for the background) and not the person detected is ignored. This neglect of background and foreground focuses data within the depth threshold of the monitored region of interest. Alternatively, the "region of interest" may include different parts of the scene, eg the foreground to see where the person is in later recognition steps, and may be enlarged or reduced as system requirements dictate. . In general, the region of interest is applied to any cut-out of the scene of focus on which to perform the person tracking.

According to certain embodiments, a number of image frames (eg, N-1 and N) are obtained and compared, and in certain examples, the image frames, as discussed above, in addition to RGB (color) data plus depth information may include The image and depth information may be filtered for noise and then processed to determine whether a difference exists between the two frames. This may be done with edge detection, threshold and difference algorithms, or other image processing techniques. In certain examples, information from the depth sensors is also processed to compare image frames. The system may use the changes between image frames, particularly the detected changes in the position or orientation of the person, to track the movement of the person. In some embodiments, change detection may be limited to a region of interest to increase processing speed.

In one embodiment, when a region of interest is determined, the extrinsic and extrinsic and A "point cloud" is created using the intrinsic parameters. In one embodiment, a point cloud library may be used. Object shape and location information generated from the point cloud library is used to identify and track a person in three dimensions using edge detection, color detection, object recognition, and/or other algorithms to determine the presence of a person in a scene. For example, if the object information is in the shape of a human, the process continues to track the person. However, if size, shape, or other appearance information indicates that the object is not a person, subsequent image frames may be analyzed until a person is detected. In some examples, images captured by video cameras 220 may include more than one person. Accordingly, the process may compare the expected features and/or appearance attributes of the person being tracked to those detected in the image frames to continue tracking the exact person.

As discussed above, central processing unit 150 may merge images obtained from video cameras 220 into a map to track the identified people as they have traveled through the business enterprise. . In certain examples, application software running on mobile device 140 is configured to display a map or similar map view or virtual layout of a floor plan of business enterprise 110 , such that the person being tracked can view its location within the business enterprise. can make it The central processing unit 150 executes the commands - by the networking device 240 - to transmit RF signals conveying the updated location of the tracked person, which can be determined using image processing techniques as discussed above. It can transmit to RF transmitters 230 . The mobile device 140 - with its RF receiver - receives these signals, and updates the person in application software that can show the person's location within the virtual layout of the business enterprise 110 displayed on the mobile device 140 . registered location.

3 shows an example display 300 of a top view of a business enterprise 110 with passageways 210 that may appear on the screen of a mobile device 140 ( FIG. 1 ) carried by a person being tracked. . In this example, arrow 310 indicates the current location of the person being tracked. As discussed above, the personalized navigation system 100 , and application software running on the mobile device 140 , may be configured to guide the tracked person through the business enterprise based on information received along with the identifier. can For example, if a shopping list is provided, the personalized navigation system 100 can access information from the business enterprise 110 that identifies where items on the shopping list are located in the aisles 210 . . In one example, central processing unit 150 ( FIGS. 1 , 2 ) examines the shopping list obtained by initial RF communication with mobile device 140 , and stores the locations of all items within business enterprise 110 . access the database The central processing unit may be further configured to match the descriptions of the items on the shopping list with product SKUs or other identifying information in the database to obtain the location of each item in the business enterprise 110 . The database may be part of the personalized navigation system 100 obtained from the operator of the business enterprise 110 and updated regularly, or the database may be maintained by the operator of the business enterprise and personalized via the cloud 250 . may be accessed by a navigation system (eg, by central processing unit 150 ).

In FIG. 3 , directional dashed lines 320 identify a proposed route through the business enterprise 110 . After the central processing unit 150 examines the shopping list and identifies where desired items 330 on the shopping list are located within the business enterprise 110, a route through the business enterprise to efficiently obtain the items. (320) is proposed. In one embodiment, the route is based on the shortest distance to obtain all items. Another route may be one in which a person initially directs perishable or refrigerated items, leaving perishable or refrigerated items at the end of the route. Central processing unit 150 transmits map and item location information to mobile device 140 via RF transmitters 230 ( FIG. 2 ). Application software running on mobile device 140 displays route 320 based on items on the tracked person's shopping list. To complement the positioning information and calibration provided by video cameras, the mobile device 140 may have inertial sensors. Techniques for improving positioning information using inertial sensing are described in U.S. Patent No. 8,957,812, issued February 17, 2015, entitled "Position Tracking System and Method using Radio Signals and Inertial Sensing," The entirety of the US patents is incorporated herein by reference. In other examples, additional RF nodes (similar to RF nodes 120 ) with RF transmit/receive capability may be used to supplement the positioning information provided by video cameras and to track the personalized navigation system 100 . It can be used to improve abilities.

In addition to identifying the desired items 330 , the central processing unit 150 can also select items that may be of interest, even if the item is not on the shopping list when a person's current location approaches the item's location. , can notify the person. Such advertisements may be based, for example, on a person's shopping history. As discussed above, in certain examples, the information provided from the mobile device 140 to the RF node 120 (and thus to the central processing unit 150 ) may include a service request. Thus, in such examples, instead of or in addition to displaying the locations of the desired items 330 , the location of the service desk or other relevant information may be displayed on the map, and the route 320 may direct the person to the location. It can be configured to guide.

Referring to FIG. 4 , another aspect of the present invention is a product tracking and checkout system 400 that can track products taken by customers as they shop at a business enterprise 110 , which includes a person's shopping It can be integrated with inventory, and also eliminate the need for shoppers to wait in cashier lines. This system 400 allows customers 410 to remove any or all of products 412 , 414 on shelves 416 , put them into their cart, and products and products taken It allows the person who took them to be registered, to automatically pay for the products, and to leave the store with the purchase of the product automatically logged by the retail system without having to go through the checkout system. Aspects of this system could improve the speed and experience of the shopping process for customers and reduce cashier's cost for retailers.

For example, a retailer's expense can automatically track inventory on shelves and/or what is taken off these shelves by customers to automatically track what customers take from the stores and to manage inventory on the shelves. It can be reduced to a system in place, which is tracked by Inventory, and the ability to track products that customers have taken from shelves will help retailers by eliminating the need for cashiers or additional staff to constantly go to shelves to inspect items that need replacement and re-stocking. It is possible to improve the cost basis for Additionally, the system may update shopping lists received from the tracked person based on items discovered and taken by the tracked person, and based on the progress made by the tracked person, the displayed route 320 may be can be updated.

It is understood that variations of image processing may be used for shelf and product tracking. One aspect of system 400 includes a product recognition camera 420 facing the shelves to view the products on the shelves and products that customers have taken. The system may have one or more first shelf-facing cameras 420 having a view angle 422 focusing on the shelf to know which products are available and which products to take. However, there may also be situations where one or more shelf-focused product recognition cameras 420 may not be sufficient, where two people reach out to grab products in the same area and/or potentially, their individual product. This is because there may be times that cross arms while reaching out to catch them and/or possibly block the view of one or more product tracking cameras 420 when reaching or grabbing product on a shelf.

Accordingly, one embodiment of the system incorporates an additional outward looking (passage facing) camera 430 . Accordingly, an aspect of this embodiment of the system includes at least two cameras on an integrated arm mount 440 . The at least one first product tracking camera 420 is oriented to focus on products on the shelf and the at least one second aisle tracking camera 430 is oriented to focus on the aisle and customers shopping. Both cameras may be video cameras, both of which, as discussed above, are video cameras to be deployed throughout the business enterprise to provide full coverage of the interior of the business enterprise 110 . It may be a video camera of ones 220-1 to 220-n (generally 220). Thus, in this embodiment, the at least one camera 420 (“shelf tracking camera”) may be primarily used for product recognition on a shelf, and the at least one additional camera 430 (“aisle tracking camera”) may be used by the customer It can mainly be used to track customer skeletons to see where you are reaching.

Some advantages of this embodiment of the system 400 are that by using the at least one aisle tracking camera 430 to focus on the shopper and within the aisle, the system can be configured with the shelf-facing camera 420 or any of the other video cameras 220 . It is possible to eliminate any occlusion problems from a shopper standing in front of the video camera of In addition, the combination of the first shelf-facing camera 420 and the second aisle-facing camera 430 may also ensure that when two shoppers reach the same area for products, the cameras do not confuse the item taken; Crossing arms or occluding the camera could potentially prevent the system from billing the wrong customer for the item taken.

Aspects of this embodiment of the system 400 with dual cameras provide shopper registration, shopper movement tracking, and product identification on retail shelves, inventory, from at least one first camera 420 and at least one second camera 430 . may include accomplishing a number of functions, including tracking, and monitoring the quantity of products on the shelf.

Typically, multiple people may be entering and/or exiting the entrance of the enterprise at any given moment. Further, more than one of them may be operating an RF transmitting mobile device at the moment, and an RF node of the personalized navigation system located near the entrance may be in communication with a plurality of people. Additionally, others passing through or near the entrance at the moment may not be carrying a mobile device or may have their mobile devices turned off. The personalized navigation systems described herein are configured to distinguish between those participating in personalized navigation and between participants and non-participants. Unlike participants, non-participants are not interacting via RF communication with a personalized navigation system, but, like participants, non-participants can still be optically tracked through the enterprise.

It is understood that aspects of cameras and systems may also include color sensing, comparison, and depth sensing, which may be achieved, for example, with infrared sensing. The first shelf tracking camera 420 may register the position of products and recognize actual products on the shelf using either or both of color and depth sensing. A second aisle tracking camera 430 may use depth sensing to perform bone tracking to determine where the customer is reaching. The item actually in the customer's hand provided by the shelf camera 420 providing product identification and detection of what was taken (from the shelf), and the position of the person's arm in relation to the item, and the aisle camera 430 when picked up. By this, a confirmation of which customer has selected which product on the shelf is achieved. The fusion of the functions of these two cameras provides a much more robust way to see what has been pulled from the shelf and by which shopper.

5 depicts one embodiment of a personalized navigation system 100 configured to differentiate between individuals passing simultaneously or side-by-side an entrance of a business enterprise 110 . Located near the entrance are one or more RF nodes 120 - 1 , 120 - 2 (typically 120 ) and one or more cameras 220 (only one is shown to simplify the drawing). Each RF node 120 has a corresponding RF receiver antenna 130-1, 130-n (typically 130) and an RF transmitter (not shown). In one embodiment, each RF node 120 has an antenna array composed of two or more antennas (or antenna elements) with known spatial separation(s). Timing differences of the RF signal received at least at each of the receiver antennas are used by the RF node to estimate the angle of arrival of the RF signal. An example of a multiplexing technique for multiplexing the antennas of an antenna array is described in US Pat. No. 8,749,433, issued Jun. 10, 2014, entitled "Multiplexing Receiver System," which is incorporated herein by reference in its entirety. do.

Each RF node 120 and each camera 220 are in a fixed location and are in communication with a controller 150 , which controller is in communication with the relative positions of each such electronic equipment (ie, between the cameras 220 , The distance between the RF nodes 120, each camera 220 and each RF node, and the pan angle and tilt angle) are known. Each camera 220 has a field of view 510 covering the area near the entrance to the enterprise. As described above, the controller 150 is configured to detect a person in the image captured by the camera(s) 220 , and determine which person was transmitted by the person's mobile device and was recently received by the RF node. configured to associate with an identifier.

As shown in Figure 5, multiple people 500-1, 500-2, and 500-3 (typically 500) are (whether entering or leaving) at the same time (whether entering or leaving) near the enterprise entrance. Individuals 500-1 and 500-2 carry RF transmitting mobile devices 140-1 and 140-2, respectively. Each of the mobile devices 140-1 and 140-2 is running application software that provides personalized navigation of the business enterprise, as described above, and uses a wireless communication technology such as Bluetooth®, Wi-Fi, or NFC. to communicate with one or more of the RF nodes 120-1 and 120-2. Such application software may be running in the background, ie, the individual does not need to intentionally activate the application to handshake with the RF node; The application is broadcasting or listening for join requests in the background. By running the application in the background, the mobile device 140 automatically connects with each RF node 120 when it comes within its wireless communication range. To synchronize communications, each mobile device 140 handshakes with each RF node 120 with which it is communicating. In this example, individual 500 - 3 is not carrying or running a mobile device in RF communication with RF node 120 . Camera(s) 220 captures images including all these individuals 500-1, 500-2, 500-3; That is, the controller can detect all individuals 500-1, 500-2, 500-3 in a single captured image. As described herein, the controller 150 determines which received identifier corresponds to which person in the captured image, and which detected person the RF mobile device communicates with the RF node 120 . is configured to determine whether it has or is not using

6 depicts one embodiment of a process 600 for distinguishing multiple people located at the entrance of a business enterprise, at least one of which is carrying and operating an RF transmitting mobile device. For illustrative purposes, consider that individuals 500 - 2 and 500 - 3 pass side by side through the entrance and proceed further into the room, while individual 500 - 1 only stops within the entrance. Mobile devices 140-1, 140-2 of individuals 500-1, 500-2 communicate with RF nodes 120-1, 120-2 that transmit their identifiers to them, respectively ( At about the same time or immediately after step 602), camera 220 captures an image with all three individuals therein (step 604).

Each RF node 120 derives a received signal strength indicator (RSSI) for a wireless signal received from each mobile device 140 and associates the RSSI measurement with a corresponding identifier (step 606). . In general, the closer the mobile device is to the RF node 120, the stronger the received signal. In step 608, the RF nodes send RSSI measurements and associated identifiers, and the camera sends the captured image to the controller. The controller associates the captured image with the RSSI measurements and associated identifiers because of the near synchronicity when the camera captured the image and the RF nodes received the RF signals. The controller also knows the relative positions of the RF nodes to each other and to the camera, and keeps track of which RF node transmitted which RSSI values.

In step 610, the controller detects three individuals in the image and establishes a depth order of the detected individuals to the camera. This determination of the depth order may use depth information obtained by the camera or by a depth sensor calibrated with the camera. The controller will rearrange the depth order of the humans in the image from the perspective of each RF node or for each RF node, thus facilitating comparison of the distance order of the RF node based on its computed RSSI values. where the depth order is derived from the images captured by the camera.

In this example, the controller receives RSSIs and identifier information for only two mobile devices. The task of the controller is to match each identifier with the appropriate mobile device, ie the appropriate one of the individuals detected in the image.

From the received signal strength indicators, the controller can calculate approximate distances to RF transmission sources (ie, mobile devices). The greater the number of RF nodes that provide RSSI values, the more accurately the controller can estimate (eg, via triangulation or multilateration) the location of each mobile device relative to those RF nodes. . Alternatively, the controller may infer the longest-shortest order for mobile devices based on their corresponding RSSI values, without calculating relative position or distance estimates. The controller arranges the RF transmitting mobile devices in distance order to the RF node 120-1 and to the RF node 120-2 (step 612).

Using the known positions of the RF nodes 120 - 1 , 120 - 2 and the camera 220 and camera angles, the controller determines the image, based on the signal strength indicators provided by the RF nodes 120 . Compares the depth order of the people detected in the to the distance order(s) determined for the mobile devices (step 614 ). Based on the comparison, the controller matches the identifiers with the detected individuals (step 616).

For example, based on the example locations of individuals 500 of FIG. 5 , RF node 120-1 determines that the signal strength of mobile device 140-1 is the signal strength of mobile device 140-2. greater than, while RF node 120-2 finds the opposite. For example, the controller may determine from the RSSI values that mobile device 140-1 is approximately 6 feet away from RF node 120-1 and is approximately 9 feet away from RF node 120-2, and that mobile device 140-1 is approximately 9 feet away from RF node 120-2. It can be calculated that 2) is approximately 11 feet away from the RF node 120-1 and approximately 4 feet away from the RF nodes 120-2. Thus, from the calculated estimated distances, for each RF node the distance order for mobile devices is merged; Accordingly, the controller 150 discovers that the mobile device 140-1 is closer to the RF node 120-1 than the mobile device 140-2.

Also, in the captured image, the controller can detect three people and determine each person's relative distance from the camera. Depth information associated with the image may help determine this relative distance. Then, since the positions of the RF nodes 120-1, 120-2 with respect to each camera and with respect to each other are known, the controller can temporarily, people) can be assigned identifiers. RSSI values, arbitrary signal strength based distance calculations, distance orders for mobile devices, and depth information associated with humans detected in the image guide these assignments. Other information can further enhance the decision-making process. For example, the entrance may be 16 feet wide, which provides a rough upper bound on the possible distance of people from the camera and from the RF node and helps establish the locations of people passing through the entrance. The controller may also take into account gaps detected in the image between adjacent individuals. Other indicators, such as orientation markers that are deliberately placed and calibrated within the field of view of the camera(s), may aid in the decision, such as over the entrance or on a wall or shelf on either side of the entrance. The anchoring features unique to the enterprise - examples of which include, but are not limited to, columns, beams, air ducts, ceiling tiles, pipes, windows, doors, shelves, and counters - may serve as orientation markers. can As another example, the RF node(s) may be within the field of view of the camera at the entrance, and thus the camera may capture the RF node in the same image capturing a plurality of individuals 500 . Then, during image processing, the controller may use the location of the RF node in the image as an orientation point to help determine the location of each detected human relative to the RF node.

Further image processing of images captured by the camera may also be employed to resolve any ambiguities. For example, the controller may attempt to detect mobile devices in both hands of the detected individuals. However, detecting a cell phone in an individual's belongings is not conclusive, as the cell phone may be turned off or it may not be communicating with a personalized navigation system. In addition, not detecting a cell phone in a person's belongings is likewise not critical, as an active RF transmitting mobile device phone can be hidden in a person's handbag or pocket. Such information, despite its inconclusive nature, can be used to increase the reliability level of accurate matching of mobile phones to humans detected in the image.

Based on all its available information, the controller can assign, for example, the mobile device closest to the RF node 120-1 to the person 500-1 closest to the camera in the image, the RF node The mobile device furthest from 120 - 1 may be assigned to the person 500 - 2 closest to the camera in the image. If ambiguity remains (ie, the level of confidence for an exact match does not exceed a threshold), eg, because not all individuals, eg, individual 500-3, are carrying RF transmitting mobile devices. , or because individuals are walking close to each other, the controller may collect additional RSSI information from multiple RF nodes 120-1 and 120-2 to determine two-dimensional range information for each of the mobile phones (step ( 618)). Also, the greater the number of RF nodes, the more accurate this range determination is. This coarse orientation information can be used to correct the location of smartphone-toting individuals within the captured images and to distinguish them from individuals who are not carrying a smartphone.

RSSI measurements from a single RF node (eg, 120-1) may be sufficient for the controller to determine the relative distance between mobile devices from the RF node and find an appropriate match for a human detected in the image. When the distances from the camera to the RF node, from the RF node to each mobile device based on the RSSI values, and from the camera to each detected human based on the depth information are known or calculated in advance, the controller controls each mobile device We can compute the approximate positions of , and match them to the humans detected in the image. As noted above, other visual orientation markers and/or subsequent visual and RF tracking (via subsequent RSSI information) may aid in the decision.

7 depicts one embodiment of another process 700 for differentiating multiple people located at the entrance of a business enterprise. At least one of the people is carrying and operating an RF transmitting mobile device running the application software described above (either as a foreground service or a background service) to communicate with the RF nodes 120 of the business enterprise. For purposes of illustration, consider that individuals 500-1, 500-2, and 500-3 (FIG. 5) are near the entrance. Mobile devices 140-1, 140-2 of individuals 500-1, 500-2 communicate with RF nodes 120-1, 120-2 that transmit their identifiers to them, respectively ( At about the same time or immediately after step 702), camera 220 captures an image with all three individuals therein (step 704).

For this process, each RF node 120 has an antenna array, estimates an angle of arrival for a wireless signal emanating from each mobile device 140 and associates the angle of arrival measurement with a corresponding identifier (step ( 706)). Each RF node 120 may determine a direction of an RF signal incident on the antenna array by measuring a time difference of arrival (TDOA) of the RF signal at an individual antenna of the antenna array. The RF node may make this TDOA measurement by measuring the phase difference in the RF signal received by each antenna or antenna element in the antenna array. From these time differences, the RF node can calculate the angle of arrival of the RF signal. As used herein, this angle of arrival corresponds to an angular position or angular line relative to the RF signal source, ie, mobile device 140 . Having the angular position data for a given mobile device at two (or more) RF nodes 120 enables calculation of an intersection point corresponding to the location of the mobile device. Approaches other than TDOA for calculating the angle of arrival are known in the art and may be used without departing from the principles described herein.

At step 708 , the RF nodes send the calculated angle of arrival (ie, angular position) measurements and their associated identifiers, and the camera sends the captured image to the controller. The controller associates the captured image with the angular position measurements and associated identifiers because of the close concurrency when the camera captured the image and the RF nodes received the RF signals. The controller also knows the relative positions of the RF nodes to each other and to the camera, and keeps track of which RF node transmitted which angular position measurements.

In step 710, the controller detects three individuals 500-1, 500-2, 500-3 in the image, and determines the angular alignment and/or depth order of the detected individuals with respect to the camera. The angular alignment corresponds to the left-right or right-left appearance of individuals in the image. For example, from the camera's point of view, individual 500-2 is between individuals 500-1 and 500-3 in the image, with individual 500-3 to the left of individual 500-2. and the individual 500-1 is to the right of the individual 500-2. Determination of the depth order may use depth information obtained by the camera or by a depth sensor calibrated with the camera.

The controller may rearrange the angular alignment and/or depth order of humans in the image, either from the perspective of each RF node or for each RF node. This rearrangement facilitates comparing the angular alignment of individuals within the image to the angular alignment of mobile devices according to the angular positions determined by the RF node.

In this example, the controller receives angular positions and identifier information for only two of the three mobile devices captured in the image, 140 - 1 , 140 - 2 ( FIG. 5 ). The task of the controller is to match each identifier with the appropriate mobile device, ie the appropriate one of the individuals detected in the image.

From the received or calculated angular positions, the controller can calculate approximate positions of the RF transmission sources (ie, mobile devices). The greater the number of RF nodes providing the angle of arrival or angular positions, the more the controller determines the positions of each mobile device relative to those RF nodes (eg, via the intersection of angular lines from two RF nodes, or three It can be estimated through triangulation or multilateration by the arrival angle lines from the above RF nodes).

Alternatively or additionally, the controller may infer a left-right (or right-left) arrangement for mobile devices based on their corresponding angular positions without calculating relative position or distance estimates (step (712)). For example, based on the angular positions calculated by the RF nodes 120 - 1 and 120 - 2 , the controller determines that the mobile device 140 - 2 is the mobile device ( 120 - 1 ) from the perspective of the RF node 120 - 1 . 140-1), and mobile device 140-2 is determined to be to the right of mobile device 140-1 from the perspective of RF node 120-2.

Using the known positions of the RF nodes 120 - 1 , 120 - 2 and the camera 220 and camera angles, the controller detects in the image, based on the angular positions provided by the RF nodes 120 . The angular arrangement or order of the people who have been identified may be compared to the angular arrangement or order determined for the mobile devices (step 714 ). Based on the comparison, the controller matches the identifiers with the detected individuals (step 716). For example, the controller may assign the mobile device 140 - 1 on the right according to the RF node 120 - 1 to the person 500 - 1 on the far right in the image, and the RF node 120 - 1 ), the mobile device 140 - 2 on the left may be assigned to the person 500 - 2 on the left of the person 500 - 1 in the image. Such matching may work without ambiguity when the number of mobile phones 140 matches the number of people detected in the image (eg, if individual 500 - 3 was not in the captured image). It also uses the angular order determined for mobile devices based on the angular positions provided by only one of the RF nodes 120 , although using more than one RF node improves precision. can be performed.

Alternatively or in addition, the controller may calculate the position of each mobile device based on the angular positions provided by the RF nodes 120 - 1 , 120 - 2 (eg, by calculating the intersection of the two angular lines). can This position has a depth component (from a reference point, such as a camera). The controller may rank the mobile devices in depth order based on the depth components of the calculated locations for the mobile devices. The controller can then match this depth order to the depth order of the individuals in the captured image.

As a support for accurately matching mobile devices to individuals (or individuals to mobile devices), the controller may cause ambiguity, particularly when individuals not using a mobile device, such as individual 500 - 3 , create ambiguity. RSSI information may be collected from multiple RF nodes 120-1, 120-2 to determine two-dimensional range information for each of the mobile phones and to match individuals to mobile devices as described above. step 718). Accordingly, the controller may use the signal strength information to supplement the match based on the angle of arrival calculations. Also, the greater the number of RF nodes, the more accurate this RSSI-based range determination is. This approximate distance information can also be used to correct the location of smartphone-toting individuals within the captured images and to associate them with an individual not carrying a smartphone (eg, individual 500-3). can be used to differentiate

As will be appreciated by those skilled in the art, aspects of the present invention may be embodied as systems, methods, and computer program products. Accordingly, aspects of the present invention may be implemented entirely in hardware, entirely in software (including but not limited to firmware, program code, resident software, microcode), or a combination of hardware and software. Aspects of the invention may also be in the form of a computer program product embodied in one or more computer readable media having computer readable program code stored thereon.

Any combination of one or more computer readable medium(s) may be used. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable medium can be a non-transitory computer-readable storage medium, examples of which include, but are not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination thereof. it doesn't happen

As used herein, a computer-readable storage medium is an instruction execution system, apparatus, device, computer, computing system, computer system, or any other inputting, processing, or outputting instruction, command, or data input. It can be any tangible medium that can contain or store a program for use by or in connection with a programmable machine or device. A non-exclusive list of specific examples of computer readable storage media includes an electrical connection having one or more wires, a portable computer diskette, a floppy disk, a hard disk, random access memory (RAM), read only memory (ROM), a USB flash drive , non-volatile RAM (NVRAM or NOVRAM), erasable programmable read-only memory (EPROM or flash memory), flash memory card, electrically erasable programmable read-only memory (EEPROM), optical fiber, portable compact disk read-only memory (CD) -ROM), DVD-ROM, optical storage device, magnetic storage device, or any suitable combination thereof.

A computer readable signal medium may include a propagated data signal having computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms including, but not limited to, electromagnetic, optical, or any suitable combination thereof. A computer-readable signal medium is any computer-readable medium that is not a computer-readable storage medium and can carry, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. can be As used herein, a computer-readable storage medium is not a computer-readable propagated signal medium or a propagated signal.

The program code may be embodied as computer readable instructions stored on or in a computer readable storage medium, for example, as source code, object code, interpreted code, executable code, or combinations thereof. Any standard or proprietary, programming or interpretive language may be used to generate the computer-executable instructions. Examples of such languages include C, C++, Pascal, JAVA, BASIC, Smalltalk, Visual Basic, and Visual C++.

Transmission of the program code embodied on a computer-readable medium may occur using any suitable medium including, but not limited to, wireless, wireline, fiber optic cable, radio frequency (RF), or any suitable combination thereof. .

The program code may run entirely on the user's device, such as mobile device 140, partly on the user's device, as a standalone software package, partly on the user's device and partly on a remote computer or entirely on a remote computer or server. have. Any such remote computer may be connected to the user's device via any type of network, including a local area network (LAN) or a wide area network (WAN), or (eg, via the Internet, an Internet service provider can be used to connect to an external computer.

Additionally, the methods of the present invention may be used in hardware-embedded electronic devices such as special purpose computers, programmed microprocessors or microcontrollers and peripheral integrated circuit element(s), ASICs or other integrated circuits, digital signal processors, discrete element circuits. Alternatively, it may be implemented on a programmable logic device such as a logic circuit, PLD, PLA, FPGA, PAL, or the like. In general, any device capable of implementing a state machine, which in turn may implement the methods proposed herein, may be used to implement the principles of the present invention.

Additionally, the disclosed methods may be readily implemented in software using an object or object-oriented software development environment that provides portable source code that may be used on a variety of computer or workstation platforms. Alternatively, the disclosed system may be partially or fully implemented in hardware using standard logic circuits or VLSI design. Whether software or hardware is used to implement systems in accordance with the present invention will depend on the speed and/or efficiency requirements of the system, the particular functionality, and the particular software or hardware systems or microprocessor or microcomputer systems being used. depend on However, the methods illustrated herein can be applied to any known or later developed systems or by one of ordinary skill in the applicable art, from the functional description provided herein and having a general basic knowledge of the computer and image processing arts. may be readily implemented in hardware and/or software using structures, devices, and/or software.

In addition, the disclosed methods may be readily implemented in software running on a programmed general purpose computer, special purpose computer, microprocessor, or the like. In such cases, the systems and methods of the present invention are implemented as a program embedded on a personal computer, such as a JAVA® or CGI script, as a resource residing on a server or graphics workstation, as a plug-in, or the like. can be Systems may also be implemented by physically integrating the systems and methods into software and/or hardware systems.

While various aspects of at least one embodiment have been described above, it will be understood that various changes, modifications, and improvements will readily occur to those skilled in the art. Such changes, modifications and improvements are intended to be part of the present invention and are intended to be within the scope of the present invention. Embodiments of the methods and apparatuses discussed herein, in application, are not limited to the arrangement of components and details of construction described in the foregoing description or illustrated in the accompanying drawings. The methods and apparatus are capable of being implemented in other embodiments and of being practiced or performed in various ways. Examples of specific implementations are provided herein for illustrative purposes only, and are not intended to be limiting. Also, the phraseology and terminology used herein are for the purpose of description and should not be regarded as limiting. The use of "including", "comprising", "having", "containing", "involving", and variations thereof herein It is intended to include the items listed below and their equivalents, as well as additional items. Reference to “or” may be construed inclusively so that any terms described using “or” may refer to any of one, more than one, and all of the described terms. Any references to front and rear, left and right, top and bottom, top and bottom, and vertical and horizontal are intended for convenience of description and refer to the systems and methods of the present invention, or components thereof, in any one It is not intended to be limited to a positional or spatial orientation. Accordingly, the foregoing description and drawings are by way of example only, and the scope of the invention should be determined from the appropriate construction of the appended claims and their equivalents.

Claims (40)

A system for tracking the location of an individual within a building, comprising:
at least one radiofrequency (RF) node disposed near an entrance to the building, the at least one radiofrequency (RF) node having an RF receiver for receiving RF signals from RF transmitting devices near the entrance to the building RF node;
at least one optical device disposed near the entrance to the building, wherein the plurality of people capture images of the plurality of people while they are near the entrance to the building; and
communicate with the at least one RF node to obtain information therefrom associated with the RF signals received by the RF receiver of the at least one RF node; communicate with the at least one optical device to capture the capture from the optical device controller to acquire the image
including,
the controller obtains, by the controller, the identity of each RF transmitting device and the angular position of the RF transmitting device relative to each RF node of the at least one RF node from the RF node of the at least one RF node based on the information associated with the RF signals,
The controller is configured to detect a plurality of humans in the image obtained by the controller from the at least one optical device, to detect an orientation marker in the image, and to determine the identity of each RF transmitting device in the image. detected in the image based on the position of the orientation marker in the image for each detected human and based on the determined angular position of the RF transmitting device relative to each RF node of the at least one RF node. tracking the location of an individual within a building, further configured to assign to one human of said plurality of humans, thereby optically identifying each individual to be located as said individual moves throughout said building; system for. 2. The method of claim 1, wherein the at least one RF node is configured to determine a relative signal strength indicator (RSSI) value for the RF signals received from each RF transmitting device; The information obtained by the controller from the RF node includes the RSSI values, and the controller determines the distance of each RF transmitting device from each RF node, the distance of each RF transmitting device from the RF transmitting device to the received by the RF node. each RF transmission from each RF node to estimate based on the RSSI values for RF signals, and when assigning the identity of each RF transmitting device to one of the plurality of humans detected in the image. A system for tracking a location of an individual within a building, configured to use the estimated distance of a device. The method of claim 1 , wherein the controller is configured to detect in the image a mobile phone carried by one of the plurality of humans detected in the image, and to detect in the image one human of the plurality of humans detected in the image. of an individual in a building, further configured to assign the identity of each RF transmitting device to one of the plurality of humans detected in the captured image based on possession of the detected mobile phone. A system for tracking your location. According to claim 1, wherein the controller,
for each RF node, arrange the RF transmitting mobile devices in an angular order based on the angular positions of the RF transmitting mobile devices from the RF node;
arrange the humans captured in the image in an angular order based on the relative positions of the plurality of humans detected in the captured image; and
When assigning the identity of each RF transmitting mobile device to one of the plurality of humans detected in the captured image, the angular order of humans detected in the captured image is compared with the angular order of the RF transmitting mobile devices. to compare
further configured, a system for tracking a location of an individual within a building. 5. The at least one RF of claim 4, wherein the controller compares the angular order of the plurality of humans in the captured image before comparing the angular order of humans detected in the image with the angular order of the RF transmitting mobile devices. A system for tracking the location of an individual within a building, further configured to arrange as to a node. The method of claim 1 , further comprising: a plurality of optical devices disposed throughout the building, wherein the controller configures each identified individual throughout the building to determine, over time, at least one of the plurality of optical devices. The system for tracking a location of an individual within a building is further configured to optically track based on detecting the identified individual in images received from a portion. The system of claim 1 , wherein the information conveyed by RF signals transmitted by a given RF transmitting mobile device comprises a shopping list. The method of claim 7, wherein the controller,
determine a route through the building based on items on the shopping list; and
transmit the route to the given RF transmitting mobile device for display on a screen of the given RF transmitting mobile device.
further configured, a system for tracking a location of an individual within a building. The method of claim 7, wherein the controller,
calculate a position for each RF transmitting mobile device based on the angular positions of the RF transmitting mobile devices received from the at least one RF node;
arrange the RF transmitting mobile devices in depth order based on the calculated positions of the RF transmitting mobile devices relative to the at least one RF node;
arrange the humans captured in the image in depth order based on the relative positions of the plurality of humans detected in the captured image; and
assign the identity of each RF transmitting mobile device to one of the plurality of humans detected in the captured image by comparing the depth order of humans detected in the image with the depth order of the RF transmitting mobile devices.
further configured, a system for tracking a location of an individual within a building. A method for tracking the location of an individual within a building, comprising:
receiving RF signals from RF transmitting mobile devices carried by a plurality of people near the entrance to the building by at least one radio frequency (RF) node disposed near an entrance to the building;
capturing an image of the plurality of people while the plurality of people are near the entrance to the building;
determining an identity of each RF transmitting mobile device and an angular position of each RF transmitting mobile device from each RF node based on information associated with the RF signals received by the at least one RF node;
detecting a plurality of humans and orientation markers in the captured image; and
of each RF transmitting mobile device based on the position of the orientation marker in the image for each human detected in the image and based on the determined angular position of each RF transmitting mobile device for each RF node. assigning the identity to one of the plurality of humans detected in the captured image, thereby optically identifying each individual to be located as the individual moves throughout the building;
A method for tracking the location of an individual within a building, comprising: 11. The method of claim 10,
determining a relative signal strength indicator (RSSI) value for the RF signals received from each RF transmitting device;
calculating an estimated distance of each RF transmitting device from each RF node based on the RSSI values for the RF signals received by the RF node from the RF transmitting device; and
using the estimated distance of each RF transmitting device from each RF node in assigning the identity of each RF transmitting device to one of the plurality of humans detected in the image.
A method for tracking the location of an individual within a building, further comprising a. 11. The method of claim 10, further comprising: detecting in the image a mobile phone carried by one of the plurality of humans detected in the image; assigning the identity of each RF transmitting device to one of the plurality of humans detected in the captured image is based on which the human detected in the image is in possession of the detected mobile phone; A method for tracking the location of an individual within a building. 12. The method of claim 11,
arranging the RF transmitting mobile devices in an angular order based on the angular positions of the RF transmitting mobile devices with respect to the at least one RF node; and
arranging the humans captured in the image in angular order based on the relative positions of the plurality of humans detected in the captured image;
further comprising: assigning the identity of each RF transmitting mobile device to one of the plurality of humans detected in the captured image comprises assigning an angular order of humans detected in the image to the RF transmitting mobile device. A method for tracking a location of an individual in a building, comprising comparing the angular order of devices. 14. The arrangement of claim 13, wherein the angular order of humans in the captured image is for the at least one RF node prior to comparing the angular order of humans detected in the captured image with the angular order of the RF transmitting mobile devices. A method for tracking the location of an individual within a building, further comprising: 11. The method of claim 10, further comprising optically tracking the location of each identified individual passing through the building by comparing successive images capturing the identified individual. How to track. 11. The method of claim 10, wherein the information associated with the RF signals transmitted by a given RF transmitting mobile device comprises a shopping list. 17. The method of claim 16,
determining a route through the building based on items on the shopping list; and
transmitting the route to the given RF transmitting mobile device for display on a screen of the given RF transmitting mobile device.
A method for tracking the location of an individual within a building, further comprising a. 11. The method of claim 10, further comprising: transmitting, from the at least one RF node, wireless signals conveying a current location of a given identified individual within the building to the RF transmitting mobile device carried by the given identified individual. further comprising a method for tracking the location of an individual within a building. 11. The method of claim 10,
calculating a position for each RF transmitting mobile device based on the angular positions of the RF transmitting mobile devices received from the at least one RF node;
arranging the RF transmitting mobile devices in depth order based on the calculated positions of the RF transmitting mobile devices relative to the at least one RF node; and
arranging the humans captured in the image in depth order based on the relative positions of the plurality of humans detected in the captured image;
further comprising: assigning the identity of each RF transmitting mobile device to one of the plurality of humans detected in the captured image comprises: assigning a depth order of humans detected in the image to the RF transmitting mobile device A method for tracking a location of an individual within a building, comprising comparing the devices to a depth order of the devices. A system for tracking the location of an individual within a building, comprising:
at least one radio frequency (RF) node disposed near an entrance to the building, the at least one radio frequency (RF) node having an RF receiver channel for receiving RF signals from a plurality of RF transmitting devices near the entrance to the building RF node;
at least one camera disposed near the entrance to the building, wherein the plurality of people capture images of the plurality of people while they are near the entrance to the building; and
communicate with the at least one RF node to obtain information therefrom associated with the RF signals received by the RF receiver channel of the at least one RF node, and communicate with the at least one camera to view the captured image therefrom A controller that obtains the identity of each RF transmitting device and the position of the RF transmitting device relative to each RF node of the at least one RF node from the RF node of the at least one RF node. to determine based on the information associated with the RF signals obtained by a controller, to detect a plurality of humans in the image obtained by the controller from the at least one camera, to detect an orientation marker in the image; and determine the identity of each RF transmitting device based on the position of the orientation marker in the image for each human detected in the image and the RF transmitting device for each RF node of the at least one RF node. and assign to one of the plurality of humans detected in the image based on the determined angular position of
A system for tracking the location of an individual within a building, comprising: A method for tracking the location of an individual within a building, comprising:
receiving, by at least one radio frequency (RF) node disposed near the entrance area, an RF signal from each of the one or more RF transmitting devices near the entrance area;
obtaining information associated with each RF signal received by the at least one RF node;
determining an identity of each RF transmitting device based on the obtained information associated with the RF signal received from the RF transmitting device;
determining a position of each RF transmitting device relative to each RF node of the at least one RF node based on the obtained information associated with the RF signal received from the RF transmitting device;
capturing an image of one or more people while the plurality of one or more people are near the entrance area;
detecting one or more humans in the image;
detecting an orientation marker in the image; and
of the RF transmitting device relative to each RF node of the at least one RF node and based on the position of the orientation marker in the image for each human detected in the image. assigning to one of the one or more humans detected in the image based on the determined position;
A method for tracking the location of an individual within a building, comprising: 22. The method of claim 21, wherein determining the position of each RF transmitting device with respect to each RF of the at least one RF node comprises an angular position of the RF transmitting device with respect to each RF node of the at least one RF node. and determining based on the angle of arrival of the RF signal received by the RF node from the RF transmitting device. 23. The method of claim 22,
arranging the one or more RF transmitting devices in an angular order based on the angular positions of the one or more RF transmitting devices with respect to the at least one RF node; and
arranging the one or more humans captured in the image in an angular order based on the relative positions of the one or more humans detected in the captured image;
further comprising: assigning the identity of each RF transmitting device to one of the one or more humans detected in the captured image comprises: the angular order of the one or more humans detected in the image A method for tracking a location of an individual within a building comprising comparing to an angular order of one or more RF transmitting devices. 23. The method of claim 22,
calculating a position for each RF transmitting device based on the angular positions of the one or more RF transmitting devices with respect to the at least one RF node;
arranging the one or more RF transmitting devices in depth order based on the calculated positions of the one or more RF transmitting devices with respect to the at least one RF node; and
arranging the one or more humans captured in the image in depth order based on the relative positions of the one or more humans detected in the captured image;
further comprising, wherein assigning the identity of each RF transmitting device to one of the one or more humans detected in the captured image comprises: a depth order of the one or more humans detected in the image. and comparing to a depth order of the one or more RF transmitting devices. 22. The method of claim 21,
determining a relative signal strength indicator (RSSI) value for the RF signal received from each RF transmitting device;
calculating an estimated distance of each RF transmitting device from each RF node based on the RSSI value for the RF signal received by the RF node from the RF transmitting device; and
using the estimated distance of each RF transmitting device from each RF node in assigning the identity of each RF transmitting device to one of the one or more humans detected in the image.
A method for tracking the location of an individual within a building, further comprising a. 22. The method of claim 21, further comprising optically tracking the location of each human detected in the image as it passes through the building by comparing successive images capturing the human. A method for tracking your location. 22. The method of claim 21, wherein the information associated with the RF signal transmitted by a given one of the one or more RF transmitting devices comprises a shopping or task list. 28. The method of claim 27,
determining a route through the building based on items on the shopping or task list; and
transmitting the route to the given RF transmitting device for display on a screen of the given RF transmitting device.
A method for tracking the location of an individual within a building, further comprising a. 22. The method of claim 21, further comprising transmitting, by the at least one RF node, RF signals conveying a current location of a given human within the building to the RF transmitting device carried by the given human. A method for tracking the location of an individual within a building. 22. The method of claim 21, further comprising: detecting in the image a mobile phone carried by one of the one or more humans detected in the image; assigning the identity of each RF transmitting device to one of the one or more humans detected in the captured image is based on which the human detected in the image is in possession of the detected mobile phone; A method for tracking the location of an individual within a building. A system for tracking the location of an individual within a building, comprising:
at least one radio frequency (RF) node disposed near an entrance area, each RF node of the at least one RF node comprising an RF receiver for receiving an RF signal from one or more RF transmitting devices near the entrance area; wherein each RF node associates an identity of each RF transmitting device with an angle of arrival of the RF signal received from the RF transmitting device;
at least one optical device disposed near the entrance area, wherein the at least one optical device captures an image of the one or more people while they are near the entrance area; and
a controller in communication with each RF node of the at least one RF node to obtain therefrom the identity of each RF transmitting device and the associated angle of arrival of the RF signal received by the RF node from the RF transmitting device; the controller communicates with the at least one optical device to obtain the captured image therefrom, the controller is configured to detect one or more humans in the image obtained by the controller from the at least one optical device, and each the one detected in the image based on each human position detected in the image and the angle of arrival of the RF signal received from the RF transmitting device by each RF node. the controller configured to assign to one human of one or more humans;
A system for tracking the location of an individual within a building, comprising: 32. The identity of claim 31, wherein the controller detects an orientation marker in the image and further based on a position of the orientation marker in the image relative to each human detected in the image. and assign to one of the one or more humans detected in the image. The method of claim 31 , wherein the controller comprises:
arrange, for each RF node, the one or more RF transmitting devices in angular order based on the angles of arrival of the RF signals received by the RF node from the one or more RF transmitting devices;
arrange the one or more humans captured in the image in an angular order based on the relative positions of the one or more humans detected in the captured image; and
When assigning the identity of each RF transmitting device to one of the one or more humans detected in the captured image, the angular order of the one or more humans detected in the captured image is assigned to the one or more RF transmissions. to compare with the angular order of the devices
further configured, a system for tracking a location of an individual within a building. The method of claim 31 , wherein the controller comprises:
calculate a position for each RF transmitting device based on the angle of arrival of the RF signal received from the RF transmitting device by each RF node of the at least one RF node;
arrange the one or more RF transmitting devices in depth order based on the calculated positions of the RF transmitting devices;
arrange the one or more humans captured in the image in depth order based on the relative positions of the one or more humans detected in the captured image; and
When assigning the identity of each RF transmitting device to one of the one or more humans detected in the captured image, the depth order of the one or more humans detected in the image is to compare with depth order
further configured, a system for tracking a location of an individual within a building. 32. The method of claim 31, wherein the at least one RF node is configured to determine a relative signal strength indicator (RSSI) value for the RF signal received from each RF transmitting device, and wherein the controller is configured to: estimate a distance of each transmitting device from each RF node of and use the estimated distance of each RF transmitting device from each RF node in assigning to one of the one or more humans detected in the image. A method for tracking the location of an individual within a building, comprising:
receiving, by at least one radio frequency (RF) node disposed near the entrance area, an RF signal from one or more RF transmitting devices near the entrance area;
determining, by each RF node of the at least one RF node, an identity of each RF transmitting device of the one or more RF transmitting devices and an angle of arrival of the RF signal received by the RF node from the RF transmitting device step;
associating, by each RF node of the at least one RF node, the identity of each RF transmitting device with the angle of arrival of the RF signal received by the RF node from the RF transmitting device;
capturing an image of the one or more people while the one or more people are near the entrance area;
detecting one or more humans in the captured image;
determining the identity of each RF transmitting device based on the position of each human detected in the image and the arrival of the RF signal received from the RF transmitting device by each RF node of the at least one RF node assigning, based on the angle, to one of the one or more humans detected in the captured image;
A method for tracking the location of an individual within a building, comprising: 37. The method of claim 36, further comprising detecting an orientation marker in the image, and assigning the identity of each RF transmitting device to one of the one or more humans detected in the captured image. is further based on the position of the orientation marker in the image for each human detected in the image. 37. The method of claim 36,
for each RF node, arranging the one or more RF transmitting devices in an angular order based on the angles of arrival of the RF signals received by the RF node from the one or more RF transmitting devices;
arranging the one or more humans captured in the image in an angular order based on the relative positions of the one or more humans detected in the captured image; and
When assigning the identity of each RF transmitting device to one of the one or more humans detected in the captured image, the angular order of the one or more humans detected in the captured image is assigned to the one or more RF transmissions. Comparing the angular order of devices
A method for tracking the location of an individual within a building, further comprising a. 37. The method of claim 36,
calculating a position for each RF transmitting device based on the angle of arrival of the RF signal received from the RF transmitting device by each RF node of the at least one RF node;
arranging the one or more RF transmitting devices in depth order based on the calculated positions of the one or more RF transmitting devices;
arranging the one or more humans captured in the image in depth order based on the relative positions of the one or more humans detected in the captured image; and
When assigning the identity of each RF transmitting device to one of the one or more humans detected in the captured image, the depth order of the one or more humans detected in the image is Steps to compare with depth order
A method for tracking the location of an individual within a building, further comprising a. 37. The method of claim 36,
determining, by the at least one RF node, a relative signal strength indicator (RSSI) value for the RF signal received from each RF transmitting device;
estimating a distance of each RF transmitting device from each RF node of the at least one RF node based on the RSSI value for the RF signal received by the RF node from the RF transmitting device; and
taking into account the estimated distance of each RF transmitting device from each RF node when assigning the identity of each RF transmitting device to one of the one or more humans detected in the image.
A method for tracking the location of an individual within a building, further comprising a.

Images