US20200380706A1

US20200380706A1 - Method, System and Apparatus for Detecting Support Structure Obstructions

Info

Publication number: US20200380706A1
Application number: US16/429,516
Authority: US
Inventors: Vlad Gorodetsky; Joseph Lam; Richard Jeffrey RZESZUTEK
Original assignee: Zebra Technologies Corp
Current assignee: Zebra Technologies Corp
Priority date: 2019-06-03
Filing date: 2019-06-03
Publication date: 2020-12-03
Also published as: US11341663B2; WO2020247271A1

Abstract

A method in an imaging controller of detecting obstructions on a front of a support structure includes: obtaining (i) a point cloud of the support structure and an obstruction, and (ii) a support structure plane corresponding to the front of the support structure; for each of a plurality of selection depths: selecting a subset of points from the point cloud based on the selection depth; detecting obstruction candidates from the subset of points and, for each obstruction candidate: responsive to a dimensional criterion being met, determining whether the obstruction candidate meets a confirmation criterion; when the obstruction candidate meets the confirmation criterion, identifying the obstruction candidate as a confirmed obstruction; and presenting obstruction detection output data including the confirmed obstructions.

Description

BACKGROUND

Environments in which objects are managed, such as retail facilities, warehousing and distribution facilities, and the like, may store such objects in regions such as aisles of shelf modules or the like. For example, a retail facility may include objects such as products for purchase, and a distribution facility may include objects such as parcels or pallets. A mobile automation apparatus may be deployed within such facilities to perform tasks at various locations. For example, a mobile automation apparatus may be deployed to capture data representing an aisle in a retail facility for use in detecting product status information. The aisle may contain other objects, however, that may reduce the accuracy of status information detected from the captured data.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

The accompanying figures, where like reference numerals refer to identical or functionally similar elements throughout the separate views, together with the detailed description below, are incorporated in and form part of the specification, and serve to further illustrate embodiments of concepts that include the claimed invention, and explain various principles and advantages of those embodiments.

FIG. 1 is a schematic of a mobile automation system.

FIG. 2 depicts a mobile automation apparatus in the system of FIG. 1.

FIG. 3 is a block diagram of certain internal components of the mobile automation apparatus in the system of FIG. 1.

FIG. 4 is a flowchart of a method of detecting support structure obstructions in the system of FIG. 1.

FIG. 5 is a diagram illustrating a point cloud to be processed via the method of FIG. 4.

FIG. 6 is a diagram illustrating an obstruction region selected from the point cloud of FIG. 5 for further processing.

FIG. 7 is a diagram illustrating an example performance of block 420 of the method of FIG. 4.

FIG. 8 is a diagram illustrating a series of selection depths employed in the method of FIG. 4.

FIG. 9 is a diagram illustrating a further example performance of block 420 of the method of FIG. 4.

FIG. 10 is a diagram illustrating a further example performance of block 420 of the method of FIG. 4.

FIG. 11 is a diagram illustrating an example performance of block 460 of the method of FIG. 4.

Skilled artisans will appreciate that elements in the figures are illustrated for simplicity and clarity and have not necessarily been drawn to scale. For example, the dimensions of some of the elements in the figures may be exaggerated relative to other elements to help to improve understanding of embodiments of the present invention.
The apparatus and method components have been represented where appropriate by conventional symbols in the drawings, showing only those specific details that are pertinent to understanding the embodiments of the present invention so as not to obscure the disclosure with details that will be readily apparent to those of ordinary skill in the art having the benefit of the description herein.

DETAILED DESCRIPTION

Examples disclosed herein are directed to a method in an imaging controller of detecting obstructions on a front of a support structure, the method comprising: obtaining (i) a point cloud of the support structure and an obstruction, and (ii) a support structure plane corresponding to the front of the support structure; for each of a plurality of selection depths: selecting a subset of points from the point cloud based on the selection depth; detecting obstruction candidates from the subset of points and, for each obstruction candidate: responsive to a dimensional criterion being met, determining whether the obstruction candidate meets a confirmation criterion; when the obstruction candidate meets the confirmation criterion, identifying the obstruction candidate as a confirmed obstruction; and presenting obstruction detection output data including the confirmed obstructions.
Additional examples disclosed herein are directed to a computing device, comprising: a memory; an imaging controller connected with the memory, the imaging controller configured to: obtain (i) a point cloud of the support structure and an obstruction, and (ii) a support structure plane corresponding to the front of the support structure; for each of a plurality of selection depths: select a subset of points from the point cloud based on the selection depth; detect obstruction candidates from the subset of points and, for each obstruction candidate: responsive to a dimensional criterion being met, determine whether the obstruction candidate meets a confirmation criterion; when the obstruction candidate meets the confirmation criterion, identify the obstruction candidate as a confirmed obstruction; and present obstruction detection output data including the confirmed obstructions.
Further examples disclosed herein are directed to a method in an imaging controller of detecting obstructions on a front of a support structure, the method comprising: obtaining a point cloud of the support structure; selecting a plurality of point subsets based on respective selection depths; detecting obstruction candidates in each point subset and, for each obstruction candidate: responsive to a decision criterion being met, determining whether the obstruction candidate meets a confirmation criterion; when the obstruction candidate meets the confirmation criterion, identifying the obstruction candidate as a confirmed obstruction; and presenting obstruction detection output data including the confirmed obstructions.
FIG. 1 depicts a mobile automation system 100 in accordance with the teachings of this disclosure. The system 100 includes a server 101 in communication with at least one mobile automation apparatus 103 (also referred to herein simply as the apparatus 103) and at least one client computing device 104 via communication links 105, illustrated in the present example as including wireless links. In the present example, the links 105 are provided by a wireless local area network (WLAN) deployed via one or more access points (not shown). In other examples, the server 101, the client device 104, or both, are located remotely (i.e. outside the environment in which the apparatus 103 is deployed), and the links 105 therefore include wide-area networks such as the Internet, mobile networks, and the like. The system 100 also includes a dock 106 for the apparatus 103 in the present example. The dock 106 is in communication with the server 101 via a link 107 that in the present example is a wired link. In other examples, however, the link 107 is a wireless link.
The client computing device 104 is illustrated in FIG. 1 as a mobile computing device, such as a tablet, smart phone or the like. In other examples, the client device 104 is implemented as another type of computing device, such as a desktop computer, a laptop computer, another server, a kiosk, a monitor, and the like. The system 100 can include a plurality of client devices 104 in communication with the server 101 via respective links 105.
The system 100 is deployed, in the illustrated example, in a retail facility including a plurality of support structures such as shelf modules 110-1, 110-2, 110-3 and so on (collectively referred to as shelf modules 110 or shelves 110, and generically referred to as a shelf module 110 or shelf 110—this nomenclature is also employed for other elements discussed herein). Each shelf module 110 supports a plurality of products 112. Each shelf module 110 includes a shelf back 116-1, 116-2, 116-3 and a support surface (e.g. support surface 117-3 as illustrated in FIG. 1) extending from the shelf back 116 to a shelf edge 118-1, 118-2, 118-3.
The shelf modules 110 (also referred to as sub-regions of the facility) are typically arranged in a plurality of aisles (also referred to as regions of the facility), each of which includes a plurality of modules 110 aligned end-to-end. In such arrangements, the shelf edges 118 face into the aisles, through which customers in the retail facility, as well as the apparatus 103, may travel. As will be apparent from FIG. 1, the term “shelf edge” 118 as employed herein, which may also be referred to as the edge of a support surface (e.g., the support surfaces 117) refers to a surface bounded by adjacent surfaces having different angles of inclination. In the example illustrated in FIG. 1, the shelf edge 118-3 is at an angle of about ninety degrees relative to the support surface 117-3 and to the underside (not shown) of the support surface 117-3. In other examples, the angles between the shelf edge 118-3 and the adjacent surfaces, such as the support surface 117-3, is more or less than ninety degrees.
The apparatus 103 is equipped with a plurality of navigation and data capture sensors 108, such as image sensors (e.g. one or more digital cameras) and depth sensors (e.g. one or more Light Detection and Ranging (LIDAR) sensors, one or more depth cameras employing structured light patterns, such as infrared light, or the like). The apparatus 103 is deployed within the retail facility and, via communication with the server 101 and use of the sensors 108, navigates autonomously or partially autonomously along a length 119 of at least a portion of the shelves 110.
While navigating among the shelves 110, the apparatus 103 can capture images, depth measurements and the like, representing the shelves 110 (generally referred to as shelf data or captured data). Navigation may be performed according to a frame of reference 102 established within the retail facility. The apparatus 103 therefore tracks its pose (i.e. location and orientation) in the frame of reference 102.
The server 101 includes a special purpose controller, such as a processor 120, specifically designed to control and/or assist the mobile automation apparatus 103 to navigate the environment and to capture data. The processor 120 is also specifically designed, as will be discussed in detail herein, to detect certain types of obstructions on the shelf modules 110. Such obstructions can be provided to product status detection mechanisms (which may also be implemented by the processor 120 itself) to improve the accuracy of such product status detection mechanisms.
The processor 120 is interconnected with a non-transitory computer readable storage medium, such as a memory 122. The memory 122 includes a combination of volatile memory (e.g. Random Access Memory or RAM) and non-volatile memory (e.g. read only memory or ROM, Electrically Erasable Programmable Read Only Memory or EEPROM, flash memory). The processor 120 and the memory 122 each comprise one or more integrated circuits. In some embodiments, the processor 120 is implemented as one or more central processing units (CPUs) and/or graphics processing units (GPUs).
The memory 122 stores computer readable instructions for performing various functionality, including control of the apparatus 103 to navigate the modules 110 and capture shelf data, as well as post-processing of the shelf data. The execution of the above-mentioned instructions by the processor 120 configures the server 101 to perform various actions discussed herein. The applications stored in the memory 122 include an obstruction detection application 123 (also simply referred to as the application 123). The application 123 may also be implemented as a suite of logically distinct applications. each implementing a suitable portion of the functionality discussed below. In general, via execution of the application 123 or subcomponents thereof and in conjunction with other components of the server 101, the processor 120 performs various actions to detect, in data representing the shelves 110 (e.g. data captured by the apparatus 103), obstructions on the shelves 110.
The memory 122 can also store data for use in the above-mentioned control of the apparatus 103, such as a repository 124 containing a map of the retail environment and any other suitable data (e.g. operational constraints for use in controlling the apparatus 103, data captured by the apparatus 103, and the like).
The processor 120, as configured via the execution of the control application 128, is also referred to herein as an imaging controller 120, or simply as a controller 120. As will now be apparent, some or all of the functionality implemented by the imaging controller 120 described below may also be performed by preconfigured special purpose hardware controllers (e.g. one or more logic circuit arrangements specifically configured to optimize the speed of image processing, for example via FPGAs and/or Application-Specific Integrated Circuits (ASICs) configured for this purpose) rather than by execution of the application 123 by the processor 120.
The server 101 also includes a communications interface 125 interconnected with the processor 120. The communications interface 125 includes suitable hardware (e.g. transmitters, receivers, network interface controllers and the like) allowing the server 101 to communicate with other computing devices—particularly the apparatus 103, the client device 104 and the dock 106—via the links 105 and 107. The links 105 and 107 may be direct links, or links that traverse one or more networks, including both local and wide-area networks. The specific components of the communications interface 125 are selected based on the type of network or other links that the server 101 is required to communicate over. In the present example, as noted earlier, a wireless local-area network is implemented within the retail facility via the deployment of one or more wireless access points. The links 105 therefore include either or both wireless links between the apparatus 103 and the mobile device 104 and the above-mentioned access points, and a wired link (e.g. an Ethernet-based link) between the server 101 and the access point.
The processor 120 can therefore obtain data captured by the apparatus 103 via the communications interface 125 for storage (e.g. in the repository 124) and subsequent processing (e.g. to detect obstructions on the shelves 110, as noted above). The server 101 may also transmit status notifications (e.g. notifications indicating that products are out-of-stock, in low stock or misplaced) to the client device 104 responsive to the determination of product status data. The client device 104 includes one or more controllers (e.g. central processing units (CPUs) and/or field-programmable gate arrays (FPGAs) and the like) configured to process (e.g. to display) notifications received from the server 101.
Turning now to FIG. 2, the mobile automation apparatus 103 is shown in greater detail. The apparatus 103 includes a chassis 201 containing a locomotive assembly 203 (e.g. one or more electrical motors driving wheels, tracks or the like). The apparatus 103 further includes a sensor mast 205 supported on the chassis 201 and, in the present example, extending upwards (e.g., substantially vertically) from the chassis 201. The mast 205 supports the sensors 108 mentioned earlier. In particular, the sensors 108 include at least one imaging sensor 207, such as a digital camera. In the present example, the mast 205 supports seven digital cameras 207-1 through 207-7 oriented to face the shelves 110.
The mast 205 also supports at least one depth sensor 209, such as a 3D digital camera capable of capturing both depth data and image data. The apparatus 103 also includes additional depth sensors, such as LIDAR sensors 211. In the present example, the mast 205 supports two LIDAR sensors 211-1 and 211-2. As shown in FIG. 2, the cameras 207 and the LIDAR sensors 211 are arranged on one side of the mast 205, while the depth sensor 209 is arranged on a front of the mast 205. That is, the depth sensor 209 is forward-facing (i.e. captures data in the direction of travel of the apparatus 103), while the cameras 207 and LIDAR sensors 211 are side-facing (i.e. capture data alongside the apparatus 103, in a direction perpendicular to the direction of travel). In other examples, the apparatus 103 includes additional sensors, such as one or more RFID readers, temperature sensors, and the like.
The mast 205 also supports a plurality of illumination assemblies 213, configured to illuminate the fields of view of the respective cameras 207. That is, the illumination assembly 213-1 illuminates the field of view of the camera 207-1, and so on. The cameras 207 and lidars 211 are oriented on the mast 205 such that the fields of view of the sensors each face a shelf 110 along the length 119 of which the apparatus 103 is traveling. As noted earlier, the apparatus 103 is configured to track a pose of the apparatus 103 (e.g. a location and orientation of the center of the chassis 201) in the frame of reference 102, permitting data captured by the apparatus 103 to be registered to the frame of reference 102 for subsequent processing.
Referring to FIG. 3, certain components of the mobile automation apparatus 103 are shown, in addition to the cameras 207, depth sensor 209, lidars 211, and illumination assemblies 213 mentioned above. The apparatus 103 includes a special-purpose controller, such as a processor 300, interconnected with a non-transitory computer readable storage medium, such as a memory 304. The memory 304 includes a suitable combination of volatile memory (e.g. Random Access Memory or RAM) and non-volatile memory (e.g. read only memory or ROM, Electrically Erasable Programmable Read Only Memory or EEPROM, flash memory). The processor 300 and the memory 304 each comprise one or more integrated circuits. The memory 304 stores computer readable instructions for execution by the processor 300. In particular, the memory 304 stores an apparatus control application 308 which, when executed by the processor 300, configures the processor 300 to perform various functions related to navigating the facility and controlling the sensors 108 to capture data, e.g. responsive to instructions from the server 101. Those skilled in the art will appreciate that the functionality implemented by the processor 300 via the execution of the application 308 may also be implemented by one or more specially designed hardware and firmware components, such as FPGAs, ASICs and the like in other embodiments.
The memory 304 may also store a repository 312 containing, for example, a map of the environment in which the apparatus 103 operates, for use during the execution of the application 308. The apparatus 103 also includes a communications interface 316 enabling the apparatus 103 to communicate with the server 101 (e.g. via the link 105 or via the dock 106 and the link 107), for example to receive instructions to navigate to specified locations and initiate data capture operations.
In addition to the sensors mentioned earlier, the apparatus 103 includes a motion sensor 318, such as one or more wheel odometers coupled to the locomotive assembly 203. The motion sensor 318 can also include, in addition to or instead of the above-mentioned wheel odometer(s), an inertial measurement unit (IMU) configured to measure acceleration along a plurality of axes.
The actions performed by the server 101, and specifically by the processor 120 as configured via execution of the application 123, to detect obstructions on the shelves 110 from captured data (e.g. by the apparatus 103) will now be discussed in greater detail with reference to FIG. 4. FIG. 4 illustrates a method 400 of detecting support structure obstructions. The method 400 will be described in conjunction with its performance in the system 100, and in particular by the server 101, with reference to the components illustrated in FIG. 1. As will be apparent in the discussion below, in other examples, some or all of the processing described below as being performed by the server 101 may alternatively be performed by the apparatus 103.
At block 405, the server 101 obtains a point cloud of the support structure. The server 101 also obtains a plane definition corresponding to the front of the support structure. In the present example, in which the support structures are shelves such as the shelves 110 shown in FIG. 1, the point cloud obtained at block 405 therefore represents at least a portion of a shelf module 110 (and may represent a plurality of shelf modules 110). The plane definition, also referred to herein as the support structure plane or the shelf plane, corresponds to the front of the shelf modules 110. In other words, the shelf plane contains the shelf edges 118.
The point cloud and shelf plane obtained at block 405 can be retrieved from the repository 124. For example, the server 101 may have previously received captured data from the apparatus 103 including a plurality of lidar scans of the shelf modules 110, and generated a point cloud from the lidar scans. Each point in the point cloud represents a point on a surface of the shelves 110, products 112, and the like (e.g. a point that the scan line of a lidar sensor 211 impacted), and is defined by a set of coordinates (X, Y and Z) in the frame of reference 102. The shelf plane may also be previously generated by the server 101 and stored in the repository 124, for example from the above-mentioned point cloud. For example, the server 101 can process the point cloud, the raw lidar data, image data captured by the cameras 207, or a combination thereof, to identify shelf edges 118 according to predefined characteristics of the shelf edges 118. Examples of such characteristics include that the shelf edges 118 are likely to be substantially planar, and are also likely to be closer to the apparatus 103 as the apparatus 103 travels the length 119 of a shelf module 110) than other objects (such as the shelf backs 116 and products 112). The shelf plane can be obtained in a variety of suitable formats, such as a suitable set of parameters defining the plane. An example of such parameters includes a normal vector (i.e. a vector defined according to the frame of reference 102 that is perpendicular to the plane) and a depth (indicating the distance along the normal vector from the origin of the frame of reference 102 to the plane).
Referring to FIG. 5, a point cloud 500 is illustrated, depicting the shelf module 110-3. The shelf back 116-3, as well as the shelf 117-3 and shelf edge 118-3 are therefore shown in the point cloud 500. Also shown in FIG. 5 is a shelf plane 504 corresponding to the front of the shelf module 110-3 (that is, the shelf plane 504 contains the shelf edges 118-3). The point cloud 500 and the shelf plane 504 need not be obtained in the graphical form shown in FIG. 5. As will be apparent to those skilled in the art, the point cloud may be obtained as a list of coordinates, and the shelf plane 504 may be obtained as the above-mentioned parameters. Example products 112 are also shown in FIG. 5, including a box 112-1, a portion of which extends forwards beyond the shelf edge 118-3.
Further, the point cloud 500 depicts an obstruction in the form of a clip strip 508 hanging from or otherwise supported by the shelf edge 118-3. The clip strip 508 may hold coupons, samples or the like, and as shown in FIG. 5, extends into the aisle from the front of the shelf module 110-3. As will be discussed below, the server 101 processes the point cloud 500 to detect the clip strip 508 (that is, to identify the position of the clip strip 508 according to the frame of reference 102). Performance of the method 400 also enables the server 101, as will be apparent in discussion below, to detect various other forms of obstacles supported in front of the shelves 110.
Referring again to FIG. 4, at block 410 the server 101 can select a set of points from the point cloud 500, corresponding to an obstruction region. As noted above, the clip strip 508 and other obstructions detectable via performance of the method 400 extend forwards, into the aisle, from the shelf modules 110. In other words, the obstructions are assumed to appear in an obstruction region in front of the shelf plane 504. To reduce the computational load imposed on the server 101 during the performance of the method 400, the server 101 can therefore select a set of points that correspond to the above-noted obstruction region. In other examples, block 410 can be omitted, and the server 101 can process the entire point cloud 500 in the remainder of the method 400.
Referring to FIG. 6, the point cloud 500 is illustrated, with an obstruction region 600 indicated. The obstruction region 600 is a region in which obstructions detectable via the method 400 (such as the clip strip 508) are expected to be present. The obstruction region 600 extends behind the shelf plane 504 by a predefined depth 602 (e.g. 2 cm, although a wide variety of other depths may also be employed). In the discussion herein, the terms “behind” or “backward” refer to locations at greater depths along the Y axis of the frame of reference 102 from the illustrated origin of the frame of reference 102. Conversely, the terms “in front” or “forward” refer to locations at smaller depths from the origin of the frame of reference 102. The obstruction region 600 also extends forward of the shelf plane 504, either by a predetermined distance, or simply to include any and all points of the point cloud 500 that are in front of the shelf plane 504. Any points behind the back surface 604 of the obstruction region 600 are ignored for the remainder of the performance of the method 400.
Selection of the set of points in the obstruction region 600 can also include eliminating any points in the point cloud 500 that extend beyond ends of an aisle of shelf modules 110. For example, the server 101 can either detect the ends of the aisle (e.g. by detecting vertical structures such as poles that typically occur at the ends of the aisle), or can retrieve known coordinates in the frame of reference 102 of the aisle ends. The obstruction region 600 is then defined to exclude points beyond the aisle ends.
Returning to FIG. 4, the server 101 then processes the selected set of points from the point cloud according to a plurality of selection depths, to detect obstacles such as the clip strip 508. In particular, at block 415, the server 101 sets a selection depth according to a coarse interval. Specifically, the selection depth set at block 415 is set by decrementing the depth of the shelf plane 504 by the coarse interval. An example performance of block 415 is illustrated at FIG. 7. Specifically, a coarse interval 700 is illustrated, and a selection depth 704 is defined as a plane parallel to the shelf plane 504 and located at a depth that is shifted forward from the shelf plane 504 by the coarse interval 700. Any points in front of the selection depth 704 are selected in the subset at block 415. A variety of coarse intervals can be employed, for example depending on the expected size of the obstructions. In the present example, the coarse interval is about 6 cm, although other coarse intervals smaller than, or larger than, 6 cm may be employed in other embodiments.
At block 420, the server 101 projects the selected subset of points to a two-dimensional image, and detects obstruction candidates in the projection. Returning to FIG. 7, a projection 708 is shown of all points in front of the selection depth 704. To detect obstruction candidates, the server performs a suitable blob detection operation (e.g. connected components analysis or the like) on the projection 708, to identify contiguous sets of points in the projection 708 that indicate the presence of a physical object. As shown in FIG. 7, the projection 708 contains two candidate obstructions 712-1 and 712-2. The server 101 may store indications of the candidate obstructions 712-1 and 712-2, such as two-dimensional bounding boxes indicating the extents of each candidate obstruction 712. As will be apparent to those skilled in the art, the candidate obstructions 712 correspond to pieces of the clip strip 508, whose forward portion has a notch 716 that results in the clip strip 508 appearing as two distinct objects at the selection depth 704.
Referring again to FIG. 4, at block 425 the server 101 determines whether candidate obstructions remain to be processed. The determination in the present example is affirmative, because the candidate obstructions 712 have not yet been processed. The performance of the method 400 therefore proceeds to block 430. At block 430, the server 101 selects the next unprocessed candidate obstruction 712 (e.g. the candidate obstruction 712-1) and determines whether the candidate obstruction satisfies a decision criterion, reflecting whether sufficient information is available to confirm or discard the obstruction candidate. The decision criterion, in the present example, is a dimensional criterion. In the present example, the dimensional criterion is a width threshold, illustrated as the width 720 in FIG. 7. The dimensional criterion reflects a predetermined assumption about the physical structure of the obstructions. In the present example, the obstructions are expected to have a relatively small width (i.e. dimension in the X axis of the frame of reference 102), in comparison to the width of the shelf module 110. As will be apparent from FIG. 7, the candidate obstruction 712-1 does not satisfy the dimensional criterion, and the determination at block 430 is therefore negative.
Following a negative determination at block 430, the server 101 returns to block 425 to determine whether any unprocessed candidate obstructions remain. In the present example, the determination is again affirmative, and at block 430, the server 101 determines that the obstruction candidate 712-2 also does not satisfy the dimensional criterion. Following assessment of the obstruction candidate 712-2, the determination at block 425 is negative, and the performance of the method 400 proceeds to block 435.
At block 435, the server 101 determines whether any selection depths remain to be processed. As noted above, the server 101 processes the selected set of points from the point cloud 500 according to a plurality of selection depths. The selection depths are defined by the above-mentioned coarse interval, as well as a fine interval. Specifically, the first selection depth is defined by decrementing (that is, moving forward) the depth of the shelf plane 504 by the coarse interval, as described above. Each subsequent selection depth is defined by incrementing (that is, moving backward) the previous selection depth by the fine interval.
Turning to FIG. 8, a set of example selection depths are illustrated, along with the shelf plane 504 and the obstruction region 600 employed to select the initial set of points at block 410. In particular, the selection depth 704 is shown as having been obtained by decrementing the depth of the shelf plane 504 by the coarse interval 700 discussed earlier. At block 415, therefore, any points with depths between the selection depth 704 and a front 800 of the obstruction region 600 are processed.
Each subsequent selection depth is set by incrementing the current selection depth by a fine interval 802. Thus, the second selection depth in the present example is a selection depth 804. When processing the point cloud 500 using the selection depth 804, any points between the selection depth 804 and the front 800 of the obstruction region 600 are processed. Further, in the present example performance of the method 400, a third selection depth 808 corresponding to the back 604 of the obstruction region 600 is also employed. Thus, when processing the point cloud 500 using the selection depth 808, any points between the selection depth 808 and the front 800 of the obstruction region 600 are processed.
Any suitable number of selection depths may be employed in the performance of the method 400, including a greater number of selection depths than the three illustrated in FIG. 8. The fine interval 802 can be predefined (e.g. as 6 mm, although larger or smaller fine intervals may also be employed in other embodiments), or can be determined dynamically by the server 101. For example, the server 101 can determine the fine interval 802 by dividing the depth between the back 604 of the obstruction region 600 and the first selection depth (e.g. 704) by a predetermined number of desired selection depths.
Other mechanisms may also be implemented to set the various selection depths employed in the performance of the method 400. For example, rather than setting the initial selection depth with the coarse interval 700 and setting subsequent selection depths with the fine interval 802, the server 101 can set each selection depth by decrementing the back 604 of the obstruction region 600 by successive multiples of the fine interval 802. In other embodiments, the selection depths can be predefined for each module 110 in the memory 122, and the server 101 therefore need only retrieve the selection depths from the memory 122.
As will now be apparent, the specific nature of the determination at block 435 may depend on the mechanism by which the selection depths are set. In the present example, at block 435 the server 101 determines whether the current selection depth (i.e. the selection depth most recently processed at block 420) is equal to or greater than the depth of the shelf plane 504. In other embodiments the server 101 can determine whether a configurable number of selection depths has been processed.
In the present example, the determination at block 435 is affirmative, because the selection depth 704 is not equal to or greater than the depth of the back 604 of the obstruction region 600. Therefore, at block 440 the server 101 expands the selected subset by setting a new selection depth according to the mechanism described above. Specifically, the updated selection depth set at block 440 is the selection depth 804 shown in FIG. 8. The subset of points to be processed has therefore been expanded to include any points with depths between the selection depth 804 and the front 800 of the obstruction region 600. The server then returns to block 420.
In a further performance of block 420, the server 101 projects the selected subset of points (which now includes both the initial subset and the additional points between the selection depths 804 and 704) to two dimensions, and detects obstruction candidates as discussed above. Turning to FIG. 9, the projection 708 is shown along with a projection 908 generated at the second performance of block 420. In the projection 908, obstruction candidates 712-3 and 712-4 are detected, corresponding respectively to the clip strip 508 and the product 112-1 shown in FIG. 5.
At block 420, the server 101 also determines whether any obstruction candidates detected at the current selection depth overlap with previously detected obstruction candidates. Thus, at block 420 the server 101 determines whether either of the obstruction candidates 712-3 and 712-4 overlap with either of the obstruction candidates 712-1 and 712-2 from the projection 708. As will be apparent, the obstruction candidate 712-3 overlaps with both the obstruction candidates 712-1 and 712-2. That is, the obstruction candidate 712-3 represents an additional portion of the clip strip 508.
When obstruction candidates overlap, as with the obstruction candidate 712-3, the server 101 updates the obstruction candidate 712-3 to indicate previous detections. The indication of previous detections can include metadata, a copy of the projection 708, or the like. In the present example, the server 101 stores an indicator 912 in association with the projection 908, indicating that the obstruction candidate 712-3 corresponds to previously detected obstruction candidates 712-1 and 712-2. In other words, overlapping obstruction candidates 712 from different selection depths are tracked as single objects throughout the performance of the method 400.
Referring again to FIG. 4, the determination at block 430 is negative for both the obstruction candidates 712-3 and 712-4, and the server 101 thus proceeds to block 435. The determination at block 435 is again affirmative, and a final selection depth is set at block 440, corresponding to the selection depth 808 shown in FIG. 8.
Turning to FIG. 10, the projections 708 and 908 are shown, as well as a projection 1008 resulting from a performance of block 420 at the selection depth 808. As seen from FIG. 8, the selection depth 808 is behind the shelf plane 504, and the shelf edges 118-3 are therefore visible in the projection 1008. The projection 1008 therefore includes detected obstruction candidates 712-5 and 712-6 that include the shelf edges 118-3 as well as the clip strip 508 and the product 112-1, respectively. The server 101 also stores indications 916 and 920 as shown in FIG. 10, indicating previous detections of overlapping obstruction candidates.
The determination at block 430 for each of the obstruction candidates 712-5 and 712-6 is affirmative, because the widths of the obstruction candidates 712-5 and 712-6 both exceed the width threshold 720. The server 101 therefore proceeds to block 445 for each of the obstruction candidates 712-5 and 712-6. At block 445, the server 101 determines whether the obstruction candidate meets a confirmation criterion. Specifically, in the present embodiment the server 101 determines whether the obstruction candidates 712-5 and 712-6 have been detected at a threshold number of previous selection depths.
The obstruction candidate 712-5, according to the indicator 916, has been detected at two previous selection depths (the selection depths 704 and 804). The obstruction candidate 712-6, on the other hand, has been detected at only one previous selection depth, as shown in the indicator 920. Assuming the threshold number of previous detections is two, the determination at block 445 is therefore affirmative for the obstruction candidate 712-5, and negative for the obstruction candidate 712-6.
Following a negative determination at block 445, the server 101 discards the obstruction candidate 712-6, as well as any stored earlier candidates corresponding to the candidate 712-6 (i.e. the candidate 712-4 in the present example). Following an affirmative determination at block 445, however, the server 101 confirms the obstruction candidate. In particular, the server 101 retrieves the bounding box or other indication of the previous detection corresponding to the candidate 712-5 (so as to not include the shelf edge 118-3 in the bounding box), and labels the bounding box as a confirmed obstruction.
Following the performance of blocks 450 and 455, and negative determinations at block 425 and 435, the server 101 proceeds to block 460. At block 460 the server 101 stores the confirmed obstruction candidates in the memory 122, and may also present, as output of the obstruction detection process, the confirmed obstruction candidates to another computing device, another application executed by the server 101, or the like.
Storing the confirmed obstruction candidates includes converting the two-dimensional bounding boxes obtained from the projections discussed above into three-dimensional bounding boxes according to the frame of reference 102. Conversion of the two-dimensional projections into three-dimensional bounding boxes can include, for example, generating a three-dimensional bounding box having a rear face at a depth corresponding to the final obstruction candidate before the dimensional criterion was satisfied at block 430, and a forward face at a depth corresponding to the first detection of the obstruction. Thus, in the present example, a three-dimensional bounding box is generated for the obstruction candidates 712-1, 712-2 and 712-3 with a rear face at the selection depth 804 and a forward face at the selection depth 704.
In other examples, generation of the three-dimensional representations of confirmed obstruction candidates is performed by retrieving the three-dimensional coordinates of points corresponding to the obstruction candidates 712-1, 712-2 and 712-3, and fitting a bounding box to those points. FIG. 11 illustrates an example three-dimensional bounding box 1100 indicating the position of the obstruction candidates 712-1, 712-2 and 712-3 (which corresponds to the position of the clip strip 508 as shown in FIG. 5).
As will now be apparent, the repeated performance of blocks 420, 425, 430, 445, 450 and 455 for a plurality of selection depths results in candidate obstructions at each selection depth either being labelled as a confirmed obstruction, discarded, or stored as neither confirmed nor discarded (for further evaluation at the next selection depth).
In some embodiments, additional confirmation criteria can be applied instead of, or in addition to, the number of detections assessed at block 445 to determine whether obstruction candidates are confirmed or discarded. For example, a minimum height threshold (i.e. a dimension along the Z axis of the frame of reference 102) can be specified following an affirmative determination at block 445, such that obstruction candidates that do not meet the minimum height are discarded. Such a minimum height threshold can also occur instead of block 445, such that a candidate obstruction meeting the minimum height threshold is confirmed regardless of the number of times the candidate obstruction was detected. In yet additional embodiments, the predetermined obstruction criteria include one or more of the following: a predetermined obstruction size range (e.g., maximum and minimum obstruction dimensions), a predetermined obstruction shape (e.g., a shape corresponding to a clip strip or other expected obstructions in front of the shelf), a predetermined orientation and/or range of orientations of the obstruction (e.g., maximum and minimum values corresponding to an orientation of expected obstructions with respect to one or more surfaces of the shelf, such as with respect to the shelf edge and/or back of the shelf), among others. In further embodiments, other decision criteria can be employed at block 430, instead of or in addition to the above-mentioned dimensional criterion. For example, in another embodiment the determination at block 430 is affirmative if either the dimensional criterion is met or if no further selection depths remain to be processed. That is, even if the dimensional criterion is not met by a candidate obstruction, the server 101 proceeds to block 445 to confirm or discard the candidate obstruction.
In the foregoing specification, specific embodiments have been described. However, one of ordinary skill in the art appreciates that various modifications and changes can be made without departing from the scope of the invention as set forth in the claims below. Accordingly, the specification and figures are to be regarded in an illustrative rather than a restrictive sense, and all such modifications are intended to be included within the scope of present teachings.
The benefits, advantages, solutions to problems, and any element(s) that may cause any benefit, advantage, or solution to occur or become more pronounced are not to be construed as a critical, required, or essential features or elements of any or all the claims. The invention is defined solely by the appended claims including any amendments made during the pendency of this application and all equivalents of those claims as issued.
Moreover in this document, relational terms such as first and second, top and bottom, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. The terms “comprises,” “comprising,” “has”, “having,” “includes”, “including,” “contains”, “containing” or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises, has, includes, contains a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by “comprises . . . a”, “has . . . a”, “includes . . . a”, “contains . . . a” does not, without more constraints, preclude the existence of additional identical elements in the process, method, article, or apparatus that comprises, has, includes, contains the element. The terms “a” and “an” are defined as one or more unless explicitly stated otherwise herein. The terms “substantially”, “essentially”, “approximately”, “about” or any other version thereof, are defined as being close to as understood by one of ordinary skill in the art, and in one non-limiting embodiment the term is defined to be within 10%, in another embodiment within 5%, in another embodiment within 1% and in another embodiment within 0.5%. The term “coupled” as used herein is defined as connected, although not necessarily directly and not necessarily mechanically. A device or structure that is “configured” in a certain way is configured in at least that way, but may also be configured in ways that are not listed.
It will be appreciated that some embodiments may be comprised of one or more specialized processors (or “processing devices”) such as microprocessors, digital signal processors, customized processors and field programmable gate arrays (FPGAs) and unique stored program instructions (including both software and firmware) that control the one or more processors to implement, in conjunction with certain non-processor circuits, some, most, or all of the functions of the method and/or apparatus described herein. Alternatively, some or all functions could be implemented by a state machine that has no stored program instructions, or in one or more application specific integrated circuits (ASICs), in which each function or some combinations of certain of the functions are implemented as custom logic. Of course, a combination of the two approaches could be used.
Moreover, an embodiment can be implemented as a computer-readable storage medium having computer readable code stored thereon for programming a computer (e.g., comprising a processor) to perform a method as described and claimed herein. Examples of such computer-readable storage mediums include, but are not limited to, a hard disk, a CD-ROM, an optical storage device, a magnetic storage device, a ROM (Read Only Memory), a PROM (Programmable Read Only Memory), an EPROM (Erasable Programmable Read Only Memory), an EEPROM (Electrically Erasable Programmable Read Only Memory) and a Flash memory. Further, it is expected that one of ordinary skill, notwithstanding possibly significant effort and many design choices motivated by, for example, available time, current technology, and economic considerations, when guided by the concepts and principles disclosed herein will be readily capable of generating such software instructions and programs and ICs with minimal experimentation.
The Abstract of the Disclosure is provided to allow the reader to quickly ascertain the nature of the technical disclosure. It is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. In addition, in the foregoing Detailed Description, it can be seen that various features are grouped together in various embodiments for the purpose of streamlining the disclosure. This method of disclosure is not to be interpreted as reflecting an intention that the claimed embodiments require more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive subject matter lies in less than all features of a single disclosed embodiment. Thus the following claims are hereby incorporated into the Detailed Description, with each claim standing on its own as a separately claimed subject matter.

Claims

1. A method in an imaging controller of detecting obstructions on a front of a support structure, the method comprising:

obtaining (i) a point cloud of the support structure and an obstruction, and (ii) a support structure plane corresponding to the front of the support structure;

for each of a plurality of selection depths:

selecting a subset of points from the point cloud based on the selection depth;

detecting obstruction candidates from the subset of points and, for each obstruction candidate:

responsive to a dimensional criterion being met, determining whether the obstruction candidate meets a confirmation criterion;

when the obstruction candidate meets the confirmation criterion, identifying the obstruction candidate as a confirmed obstruction; and

presenting obstruction detection output data including the confirmed obstructions.

2. The method of claim 1, wherein determining whether the obstruction candidate meets the confirmation criterion includes at least one of:

determining whether the obstruction candidate exceeds a minimum obstruction dimension;

determining whether the obstruction candidate has a predetermined obstruction shape; and

determining whether the obstruction candidate has a predetermined obstruction orientation.

3. The method of claim 1, wherein determining whether the obstruction candidate meets the confirmation criterion includes determining whether the obstruction candidate has been detected at a threshold number of previous selection depths.

4. The method of claim 3, further comprising:

when the obstruction candidate has not been detected at the threshold number of previous selection depths, discarding the obstruction candidate.

5. The method of claim 1, further comprising:

responsive to the dimensional criterion not being met, storing the obstruction candidate in the memory for evaluation at a subsequent selection depth.

6. The method of claim 1, further comprising:

selecting a set of points from the point cloud corresponding to an obstruction region;

wherein the subset of points is selected from the obstruction region.

7. The method of claim 1, further comprising setting the selection depths by:

decrementing a depth of the support structure plane by a coarse interval to set a first selection depth; and

incrementing the first selection depth by a fine interval to set a second selection depth.

8. The method of claim 1, wherein selecting the subset of points includes selecting the points having depths smaller than the selection depth.

9. The method of claim 1, wherein detecting obstruction candidates comprises:

generating a two-dimensional projection of the selected subset of points;

detecting contiguous sets of points in the projection; and

generating a bounding box corresponding to each contiguous set.

10. The method of claim 7, wherein detecting obstruction candidates further comprises:

determining whether the bounding box overlaps with a previously detected obstruction candidate; and

when the bounding box overlaps with a previously detected obstruction candidate, storing an indication of the previously detected obstruction candidate with the bounding box.

11. The method of claim 1, wherein the dimensional criterion is a threshold width.

12. A computing device, comprising:

a memory;

an imaging controller connected with the memory, the imaging controller configured to:

obtain (i) a point cloud of the support structure and an obstruction, and (ii) a support structure plane corresponding to the front of the support structure;

for each of a plurality of selection depths:

select a subset of points from the point cloud based on the selection depth;

detect obstruction candidates from the subset of points and, for each obstruction candidate:

responsive to a dimensional criterion being met, determine whether the obstruction candidate meets a confirmation criterion;

when the obstruction candidate meets the confirmation criterion, identify the obstruction candidate as a confirmed obstruction; and

present obstruction detection output data including the confirmed obstructions.

13. The computing device of claim 12, wherein the imaging controller is configured, in order to determine whether the obstruction candidate meets the confirmation criterion, to at least one of:

determine whether the obstruction candidate exceeds a minimum obstruction dimension;

determine whether the obstruction candidate has a predetermined obstruction shape; and

determine whether the obstruction candidate has a predetermined obstruction orientation.

14. The computing device of claim 12, wherein the imaging controller is configured, in order to determine whether the obstruction candidate meets the confirmation criterion, to determine whether the obstruction candidate has been detected at a threshold number of previous selection depths.

15. The computing device of claim 12, wherein the imaging controller is further configured to:

when the obstruction candidate has not been detected at the threshold number of previous selection depths, discard the obstruction candidate.

16. The computing device of claim 12, wherein the imaging controller is further configured to:

responsive to the dimensional criterion not being met, store the obstruction candidate in the memory for evaluation at a subsequent selection depth.

17. The computing device of claim 12, wherein the imaging controller is further configured to:

select a set of points from the point cloud corresponding to an obstruction region;

wherein the subset of points is selected from the obstruction region.

18. The computing device of claim 12, wherein the imaging controller is further configured, in order to set the selection depths, to:

decrement a depth of the support structure plane by a coarse interval to set a first selection depth; and

increment the first selection depth by a fine interval to set a second selection depth.

19. The computing device of claim 12, wherein the imaging controller is further configured, in order to select the subset of points, to select the points having depths smaller than the selection depth.

20. The computing device of claim 12, wherein the imaging controller is further configured, in order to detect obstruction candidates, to:

generate a two-dimensional projection of the selected subset of points;

detect contiguous sets of points in the projection; and

generate a bounding box corresponding to each contiguous set.

21. The computing device of claim 20, wherein the imaging controller is further configured, in order to detect obstruction candidates, to:

determine whether the bounding box overlaps with a previously detected obstruction candidate; and

when the bounding box overlaps with a previously detected obstruction candidate, store an indication of the previously detected obstruction candidate with the bounding box.

22. The computing device of claim 12, wherein the dimensional criterion is a threshold width.

23. A method in an imaging controller of detecting obstructions on a front of a support structure, the method comprising:

obtaining a point cloud of the support structure;

selecting a plurality of point subsets based on respective selection depths;

detecting obstruction candidates in each point subset and, for each obstruction candidate:

responsive to a decision criterion being met, determining whether the obstruction candidate meets a confirmation criterion;

presenting obstruction detection output data including the confirmed obstructions in a memory.