WO2020034044A1 - System and method for predicting delivery parameters in an intermodal logistics network - Google Patents

Abstract

A system and method are provided for providing delivery options. The method includes interfacing a supply-chain management system with a retailer user interface and detecting a request from the retailer user interface to schedule a delivery of one or more items through an integrated supply chain network coordinated by the supply-chain management system. The method also includes, based on the request, determining from the integrated supply chain network, at least one delivery option for at least one delivery date, each delivery option comprising a route, one or more carriers for the route, and a time window. The method also includes, for each delivery option, use a prediction engine and at least one data model to compute at least one delivery prediction parameter indicative of a likelihood of success for completing the route within the time window using the one or more carriers for the route, the at least one data model having been generated using historical delivery data. The method also includes providing, in response to the request, each of the at least one delivery option augmented with the at least one delivery prediction parameter to enable the retailer user interface to display the at least one delivery option augmented with the at least one delivery prediction parameter in a user interface element enabling selection of a delivery date for the one or more items.

Description

SYSTEM AND METHOD FOR PREDICTING DELIVERY PARAMETERS IN AN

INTERMODAL LOGISTICS NETWORK

CROSS-REFERENCE TO RELATED APPLICATION(S)

[0001] This application claims priority to U.S. Provisional Patent Application No.

62/719,477 filed on August 17, 2018, the contents of which are herein incorporated by reference.

TECHNICAL FIELD

[0002] The following relates to predicting delivery parameters such as delivery windows and a likelihood of success, in particular using delivery predictions in an intermodal logistics network.

BACKGROUND

[0003] Currently retailers establish and utilize a supply chain that allows goods to be obtained from vendors such as manufacturers and factories, wholesalers, and distributors; distributed to warehouses within certain geographical areas; and populated in bricks and mortar retail stores or outlets or made available online. Customers wishing to purchase the goods from a retailer typically makes a purchase at the store, or through an online e- commerce site. For large items, a“last mile” delivery from the store or warehouse to the customer is normally required, e.g., for mattresses, appliances, furniture, etc. When shopping online, goods of all sizes requiring shipping, and this often occurs between a retailer’s warehouse or storeroom and the customer’s home, office, or other designated delivery location.

[0004] Many vendors typically supply many different retailers, each having separate warehouses, physical storerooms, and even warehouses for carriers that make the last mile delivery. Because of this, most retailers establish their own supply chain network. While a dedicated supply chain network has an upside in providing increased control and flexibility within the network, there are normally significant costs associated with having too little or too much capacity in the network. Moreover, consumers tend to wait additional time for their delivery because retailers or carriers may need to ship a full truck load for economic reasons. Another issue is with synchronizing other carrier loads with the ones that are expected by the consumers. These costs can translate to higher shipping costs to be paid by the customer or be absorbed by the retailer or vendor, or lost sales because delivery times are too long, to name a few.

[0005] In such supply chain networks it is common to have multiple“legs” in a delivery with some legs requiring a different mode of transportation. For example, some goods may be shipped by boat or airplane, be transferred to rail at a seaport, then be transported to a warehouse via trucks. The goods may then be transported from warehouses to a retail store or directly to the customer. Each leg in a delivery can be susceptible to service failures, which in highly connected supply chain networks, particularly intermodal networks, any failure can have an effect throughout the entire network.

[0006] It is an object of the following to address at least one of the following

disadvantages.

SUMMARY

[0007] The following can analyze data to create data models that predict service failures to determine a likelihood of a successful on time delivery in a supply chain network. The analysis can utilize machine learning techniques from historical data and can be used for real-time service prediction for establishing new deliveries and to provide insight to carriers and retailers regarding any issues with service deliveries.

[0008] In one aspect, there is provided a method of providing delivery options comprising: interfacing a supply-chain management system with a retailer user interface; detecting a request from the retailer user interface to schedule a delivery of one or more items through an integrated supply chain network coordinated by the supply-chain management system; based on the request, determining from the integrated supply chain network, at least one delivery option for at least one delivery date, each delivery option comprising a route, one or more carriers for the route, and a time window; for each delivery option, use a prediction engine and at least one data model to compute at least one delivery prediction parameter indicative of a likelihood of success for completing the route within the time window using the one or more carriers for the route, the at least one data model having been generated using historical delivery data; and providing, in response to the request, each of the at least one delivery option augmented with the at least one delivery prediction parameter to enable the retailer user interface to display the at least one delivery option augmented with the at least one delivery prediction parameter in a user interface element enabling selection of a delivery date for the one or more items.

[0009] BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Embodiments will now be described with reference to the appended drawings wherein:

[0011] FIG. 1A is a schematic diagram illustrating a simplified retail supply chain enabled by a supply chain management system as herein described. [0012] FIG. 1 B is a schematic diagram illustrating another example supply chain that includes intermodal transportation;

[0013] FIG. 1 C is a schematic diagram illustrating an example delivery route with multiple legs.

[0014] FIG. 2 is a block diagram illustrating an implementation of a consolidated supply chain network facilitated by a supply chain management system interfaced with multiple retailer channels.

[0015] FIG. 3 is a block diagram of an example configuration for a delivery prediction module utilized by the supply chain management system.

[0016] FIG. 4 is a network abstraction illustrating a centralized consolidation segment in a supply chain network having increased density at origin and destination areas.

[0017] FIG. 5 is a flow chart illustrating supply/delivery and reverse logistic channels within a consolidated supply chain network.

[0018] FIG. 6 is a flow chart illustrating operations performed in arranging a home delivery of an item purchased by a customer through a retailer interfaced with the supply chain management system.

[0019] FIG. 7 is a flow chart illustrating operations performed in providing delivery prediction data when presenting options for selecting a delivery option from a calendar user interface.

[0020] FIG. 8A is an example of a graphical user interface for a calendar.

[0021] FIG. 8B is an example of a graphical user interface for displaying a set of delivery options.

[0022] FIG. 9 is a graph illustrating a distribution of failure types.

[0023] FIG. 10A is a chart showing performance of a classifier for a customer not at home failure type.

[0024] FIG. 10B is a chart showing performance of a classifier for a stop rescheduled failure type.

[0025] FIG. 10C is a chart showing performance of a classifier for a refused by customer failure type.

[0026] FIG. 10D is a chart showing performance of a classifier for a canceled by customer failure type. [0027] FIG. 10E is a chart showing performance of a classifier for a not in stock failure type.

[0028] FIG. 10F is a chart showing performance of a classifier for example resampling methods.

[0029] FIG. 1 1 A is a feature importance chart for a customer not at home failure.

[0030] FIG. 1 1 B shows filtered association rules for the customer not at home failure exemplified in FIG. 1 1A.

[0031] FIG. 12A is a feature importance chart for a stop rescheduled failure.

[0032] FIG. 12B shows filtered association rules for the stop rescheduled failure exemplified in FIG. 12A.

[0033] FIG. 13A is a feature importance chart for a refused by customer failure.

[0034] FIG. 13B shows filtered association rules for the refused by customer failure exemplified in FIG. 13A.

[0035] FIG. 14A is a feature importance chart for a canceled by customer failure.

[0036] FIG. 14B shows filtered association rules for the canceled by customer failure exemplified in FIG. 14A.

[0037] FIG. 15A is a feature importance chart for a not in stock failure.

[0038] FIG. 15B shows filtered association rules for the not in stock failure exemplified in FIG. 15A.

[0039] FIG. 16A provides a visual depiction of geographical location of services in a training dataset without sampling.

[0040] FIG. 16B provides a visual depiction of geographical location of services in a training dataset after over-sampling with SMOTE.

[0041] FIG. 16C provides a visual depiction of geographical location of services in a training dataset after random under-sampling.

[0042] FIG. 16D provides a visual depiction of geographical location of services in a training dataset after NearMiss-3 under-sampling.

DETAILED DESCRIPTION

[0043] A delivery prediction module is provided that utilizes and leverages delivery data, including both current and historical data, to predict delivery parameters in supply chain networks. For example, data models can be built and analyzed to predict service failures to enable data and statistics to be presented to retailers and customers that are indicative of a likelihood of a successful delivery for a given time window or to enable such a time window to be widened.

[0044] The following illustrates an example of a system and related methods, software platforms, and user interfaces that expose, abstract, and integrate multiple supply chain network elements from different parties into a single supply chain and delivery network. The single supply chain network enables the density of goods handled at any given edge in the network to be increased by making excess capacity available across what would normally be separate and distinct supply chain and delivery networks. The system exemplified herein can utilize the delivery prediction module to provide more and better data to customers and retailers when selecting a delivery option.

[0045] The following system enables multiple retailers to utilize common consolidation and warehousing facilities, as well as common delivery and shipping

networks/routes/trucks/personnel, to increase the efficiencies of all networks. In this way, shipping costs can be minimized to the benefit of consumers enabling retailers to compete on product offerings, services, experience, and usability rather than shipping charges.

Moreover, exposing and providing access to capacities in other networks reduces the reliance of any given retailer on a particular transportation or shipping company, e.g., in the event of surges in demand, labor disruptions, etc. This can enable delivery speeds to be increased, for example to achieve same day or next day deliveries that may not otherwise be possible.

[0046] The system can therefore integrate a plurality of supply chain networks, the system comprising interfaces to vendors, consolidation centers, warehouses, retail facilities, and last mile delivery companies to enable multiple retailers to use at least one facility or transportation vehicle from another supply chain or delivery network to utilize excess capacities.

[0047] The delivery prediction module described below can be used to predict and explain service failures in supply-chain networks, for example, among last-mile pickup and delivery services to customers. The delivery prediction module 15 can analyze datasets of services using, for example: (1) supervised classification with Random Forests, and (2) association rules. The classifier described herein has been found to reach an average sensitivity of 0.7 and specificity of 0.7 for a set of five (5) studied types of failure. Association rules reassert the importance of confirmation calls to prevent failures due to customers not at home, show the importance of the time window size, slack time, and geographical location of the customer for the other failure types, and highlight the effect of the retailer company on several failure types. To reduce the occurrence of service failures, the data models described herein could be coupled to optimizers, or used to define counter-measures to be taken by human dispatchers. As noted above, the data models can also be used to analyze legs in a delivery route (on a carrier-by-carrier basis if applicable) to predict service delivery likelihood of success for each leg and/or for an overall delivery route, which can be integrated into a supply chain management system such as that shown herein.

[0048] Turning now to the figures, FIG. 1A illustrates a configuration that exposes, abstracts, and integrates multiple supply chain network elements from different parties into a single supply chain and delivery network. In this example, multiple vendors 10 ship or deliver goods to one or more consolidation centers 12 interfaced with or controlled by a centralized supply chain management (SCM) system 14. The SCM system 14 acts as a layer of abstraction for various disparate supply chain network elements (nodes) to allow for efficiencies to be built into individual retail channels. This is done by not only coordinating the movement and storage of goods within the consolidation facilities (typically referred to as the mid-mile of the network) separate from any given retail or vendor channel, but also coordinating the last-mile deliveries from the retailers (or consolidation centers) directly to the consumers 16. In this way, the SCM system 14 can increase the density of goods in any given edge of the network to enable retail stores 18 and warehouses 20 (retailers 21 generally) to provide reduced-cost deliveries to the consumers 16.

[0049] While coordinating the movement of goods between vendors 10 and retailers 21 , 20, the SCM system 14 can provide user interfaces within the delivery and shipping environment, the retail environment, and to the customer to allow the integrated supply chain and delivery flow to appear seamless to the customer. This is enabled using APIs and other interfaces between the SCM system 14 and the various entities in the supply and delivery chains to offload the coordination and integration of the process to the SCM system 14.

[0050] FIG. 1 B illustrates a supply chain in one example, wherein the mid-mile portion can utilize intermodal transportation, such as air, truck, rail, and boat. In this example, the first mile includes raw materials and parts that are delivered to manufacturer(s) to generate finished product(s). As noted in FIG. 1 B, the first mile can include numerous manufacturers over several cities in complex supply chain networks. The mid mile in this example includes a consolidation center 12 and a last mile delivery warehouse 20. Also shown in FIG. 1 B is a last mile business to business (B2B) portion of the network and a las mile business to consumer (B2C) portion of the network. Additionally, it may be noted that the supply of goods may relay on manufacturing the product correctly at the beginning of the process and goods may come from multiple manufacturers or vendors to be shipped to the entity that will prepare the finished product. FIG. 1 C illustrates an example of a delivery route that includes a number of warehouses 20, a CUDS, and customer.

[0051] Turning now to FIG. 2, an implementation of the SCM system 14 is shown. The SCM system 14 is shown in this illustration as a layer, intermediary, back-end, or complementary service that is plugged into each of the stages of the supply and delivery network to blur the lines between inventory storage nodes. That is, by relying on the SCM system 14, any consolidation center, warehouse, etc. can store goods until they are ready to be placed into a last mile shipment, separate from any strict association with a particular retail channel or distribution entity. As shown in FIG. 3, the vendors 10 (typically considered the first mile), are interfaced with the SCM system 14 to have goods required by a particular retailer 21 (in this example highlighted retail store 18a) shipped to a particular consolidation center 12a.

[0052] The mid mile portion of the supply chain and delivery network is therefore also plugged into the SCM system 14 such that dedicated consolidation centers 12 are not necessarily required for each retailer’s 21 channel. This not only allows new retailers 21 to emerge without the costly up-front costs of establishing consolidation facilities, but also existing retailers 21 to maximize the usage of their existing facilities, which may already be a sunk cost. This is illustrated in FIG. 2 as a sub-network of consolidation facilities (12a, 12b, 12c in this example) that can be integrated from various sources, including SCM system- managed consolidation centers 12. Based on data acquired over time in a logistics database 32, the SCM system 14 can determine optimal numbers and locations for such consolidation centers 12 to allow, for example, collaborations or eliminations to emerge.

The logistics database 32 can also be leveraged by 3^rd parties, e.g. via 3^rd party intelligence requests 34. This allows organizations such as retailers, investors, government planners, shipping companies, etc. to use the SCM system’s datasets to optimize geographic placement of assets, fleet sizes, personnel deployment, etc.

[0053] Several retailers 21 or“retail channels” are shown operating in parallel in FIG. 2. Each retailer 21 may include its own network of bricks and mortar retail stores 18, regional warehouses 20, online e-commerce sites, etc. In addition to optimizing the flow of goods in the first and mid mile portions, it can be appreciated that the SCM system 14 can also enable retailers to eliminate redundancies in warehousing, e.g., if consolidation centers provide sufficient storage and proximity to the retail storerooms or can service direct-to- customer delivers through online purchases. In other words, the SCM system 14 can also optimize intra-channel flow of goods for any particular retailer 21 , which again may include storage and shipment of goods for multiple retailers 21 at the same time.

[0054] The SCM system 14 also integrates and coordinates the last mile portion of the supply and delivery network, by identifying a particular delivery truck 24 (24a in this example) for the goods that will flow from the particular consolidation center 12a and the retail store 18a. Further detail regarding the coordination of the last mile delivery portion can be found in co-pending U.S. Patent Application No. 15,000,899 filed on January 19, 2016 and published as U.S. 2016/0210591 , the contents of which are incorporated herein by reference. Moreover, further detail regarding the SCM system 14 can be found in PCT Patent Application No. PCT/CA2018/050810 filed on June 29, 2018 and published as WO 2019/000106, the contents of which are incorporated herein by reference.

[0055] The coordination and execution of the flow of goods can be transparent to the customer 16 by having the SCM system 14 integrated into the various retail portals provided to the customer 16 when making a purchase, arranging delivery, and tracking and completing a shipment. For example, as shown in FIG. 2, upon purchasing an item that requires or is desired to include shipping, the customer 16 is presented with an option to select a delivery data and possible one or more delivery time windows. By being integrated with the upstream portions of the supply and delivery network, the SCM system 14 can provide the retail portal with data that allows such delivery dates to be determined. It can be appreciated that various constraints on delivery dates may be imposed depending on the location of the customer, the type of goods being purchased, etc. In any event, through the integration shown in FIG. 2, the retailer 21 and/or the SCM 14 can provide the customer 16 with options that are consistent with the efficiencies built into the shipment and delivery legs of the network.

[0056] The retailer 21 after receiving the delivery date request 38 can have the SCM system 14 coordinate the flow of goods to either the store or outlet 18 for pick-up, or direct to the customer 16 using a last mile delivery. The SCM system 14 begins with the selected delivery date to back track within the system to have the vendor 10 ship the good(s) at a particular prior date which takes into account available capacities, routes and travel times, holidays, availability from manufacturers, and whether and how many consolidation and warehousing steps are required to fit within the particular retailer’s channel. By having the retailer 21 expose their supply chain network to the SCM system 14, the SCM system 14 can find the most effective way to transport the goods from, for example, a vendor to a customer delivery or pick-up location. Similar logistics are planned at this time to coordinate last mile deliveries by exposing driver fleets that are able to make the delivery.

[0057] As shown in FIG. 2, the SCM system 14 can include a delivery prediction module 15 to analyze data related to service deliveries and provide data that can be used to predict the likelihood of success for a given route, leg, or time window associated with a delivery plan and/or selected carrier(s). FIG. 3 illustrates an example of a configuration for the delivery prediction module 15. In this example, the delivery prediction module 15 includes one or more processors 40 for executing operations for the module 15. It can be appreciated that the processor 40 and other components of the delivery prediction module 15 may be provided by the SCM system 14 and are delineated as shown in FIG. 3 for illustrative purposes only. The module 15 also includes a communications module 42 to enable the module 15 to communicate with the SCM system 14 and/or other entities within the computing environment shown in FIG. 2, if applicable. The module 15 also includes a SCM system interface module 44 that may be used for specific integration with the SCM system 14. For example, the interface module 44 may provide an application programming interface (API) to enable such integration with the SCM system 14.

[0058] The delivery prediction module 15 also includes a prediction engine 48 which may be executed to perform a prediction as described in greater detail below. The prediction engine 48 may utilize or otherwise communicate with a machine learning module 50 to perform such predictions. The delivery prediction module 15 may also store one or more data models 46 used in generating delivery predictions as described later. The delivery prediction module 15 can also use a scheduling interface module 52 to coordinate its outputs with outputs provided by the SCM system 14 and/or a retailer user interface in enabling customers to be presented with results generated by the prediction engine 48 for selecting a delivery option, e.g., in a calendar interface. The delivery prediction module 15 in this example can also store other data 47 that can be determined by the prediction engine 48 or obtained by the module 15 from third party data sources. Such other data 47 can include, for example, damage reports or survey results from customers based on historical deliveries.

[0059] FIG. 4 illustrates a network diagram with increased density at the origin and destination nodes based on a coordinated consolidation node in between. This allows goods at various origin points to more efficiently channel towards shared consolidation locations, providing more efficient deployment into higher density delivery routes at the destination end. For example, a single transportation vehicle could be arranged to ship goods from multiple vendors to the same consolidation center 12. Then, deliveries from retailers 21 that require multiple goods from similar but different vendors 10 can utilize the same delivery trucks to bring down the overall cost of shipping to the customers 16.

[0060] FIG. 5 illustrates the abstracted flow of goods from line-haul to consolidation to storage/warehousing to last mile delivery to customers in a supply/delivery direction, as well as in the reverse direction at any one of these edges by way of reverse logistics. The SCM system 14 can be used to coordinate each of these legs or edges of the network transparently to the customer 16 and even the retailer 21 to provide a seamless supply chain without requiring dedicated facilities at all or any of the nodes.

[0061] The supply and delivery network can itself also include multiple layers. For example, in B2B interactions, the following layers are facilitated and coordinated:

[0062] 1) Vendor -> Consolidation center

[0063] 2) Distribution center -> Retail stores

[0064] 3) Pick-up store - Warehouse or Consolidation center

[0065] 4) Consolidation center - Home delivery

[0066] 5) Reverse Logistics for 1) to 4)

[0067] In B2C interactions, the following layers are facilitated and coordinated:

[0068] 1) Consolidation/Warehouse ^Consumer

[0069] 2) Consumer -> Warehouse

[0070] 3) Retail store -> Consumer

[0071] 4) Vendor -> Consumer

[0072] 5) Customer pick up

[0073] 6) Reverse Logistics, in particular for 1), 2), and 4).

[0074] In one example, reverse logistics can be applied to facilitate the flow of goods from the consumer or store (e.g. returns) back to consolidation centers 12 or warehouses 20 to enable those goods to be picked up in another delivery channel or transaction. This may allow for tighter delivery dates, particularly when the lead time to obtain goods from the vendor 10 is long, but goods are able to flow in reverse within the abstracted consolidation and shipping portions of the network. [0075] FIG. 6 is a flow diagram illustrating the coordination of shipping and delivery of an item purchased by a customer 16. At step 50, the SCM system 14 indicates to the retailer 21 or its online portal available delivery days. It can be appreciated that step 50 may be optional if the retailer 21 instead controls the delivery days and only requires the SCM system 14 to coordinate thereafter. However, if data is available to the SCM system 14, it can provide any constraints around how quickly an item can be delivered, by having access to the various legs of the network. As will be explained in greater detail below, the SCM system 14 can also utilize the delivery prediction module 15 to provide additional detail with respect to the likelihood of a successful delivery. At step 52, the retailer 21 provides an interface to the customer 16 with a delivery option or multiple delivery options, which can include the results generated by the delivery prediction module 15 (discussed further below). For example, when purchasing a large item such as an appliance or furniture, many customers require delivery using a last mile delivery truck. Through coordination with the SCM system 14, the retailer 21 can provide this option to allow the customer 16 to select a delivery date for the order at step 54. Based on this selection, the retailer 21 notifies the SCM system 14 at step 56 of the selected delivery date, such that the SCM 14 can work backwards to ensure that the goods are either delivered from the vendor 10 to the appropriate consolidation center 12 or warehouse 20 or moved within the mid mile portion to efficiently transfer into the last mile leg of the delivery, which can include coordinating intermodal transportation as shown in FIG. 1 B.

[0076] At this point, the customer 16 has completed the purchase and delivery arrangement from his/her perspective, and the retailer 21 coordinates with the SCM system 14 to facilitate the flow of the goods associated with the order. For example, at step 58 the SCM system 14 can arrange a vendor transfer to a consolidation center 12 and a subsequent transfer from the consolidation center 12 to a warehouse 20 or retail store 18 at step 60. The SCM system 14 can also arrange the last mile delivery to the customer 16 at step 62, that accounts for any of the other arrangements, including timing, availability, holidays, etc. The retailer 21 can also confirm the delivery date to the customer 16 at step 64, for example, if any of the arrangements in steps 58-62 require adjustment of the delivery date, or to simply provide a tracking number, delivery company info, driver info, etc. Since some or all of the steps taken by the SCM system 14 may occur after the customer 16 makes the purchase and selects the delivery date, step 64 may be required one or more times after the retail transaction occurs. For example, the customer 16 may perform step 54 at a retail store 21 after viewing some furniture and have a follow-up email sent that confirms the delivery details sometime after leaving the retail store. In the interim, the SCM system 14 coordinates and facilitates the most efficient supply and delivery chain transparently or substantially transparently to the customer 16.

[0077] In order to implement a process such as that shown in FIG. 6, the SCM system 14 can have a central analysis, integration, and coordination sub-system 30 that uses APIs into the various supply chain elements (including those shown by way of example in FIG. 6). In addition to coordinating the flow of goods, the sub-system 30 can also be used to collect and analyze data to continually improve and refine the algorithms used to plan routes, determine optimal consolidation centers 12 and locations, etc.

[0078] FIG. 7 provides an example of a workflow for utilizing the delivery prediction module 15 in steps 50 and 52 shown in FIG. 6. Turning now to FIG. 7, at step 100 the SCM system 14 can initiate the delivery prediction module 15 to provide delivery service data for a particular delivery being executed, e.g., as shown according to FIG. 6 at steps 50 and 52. In this example, step 100 may include using the prediction engine 15 and the data model(s) 46 to compute a delivery prediction parameter, as discussed in greater detail below. It can be appreciated that the delivery prediction parameter can be presented as a likelihood (e.g., percentage) of failure or success by inverting the computed result if applicable. The prediction engine 15 can determine statistics for each leg of a route and/or each carrier that is involved. This allows a granular view of the proposed delivery option to be presented to the customer and retailer. As noted above, other data 47 such as damage reports and survey results, when available, can augment the delivery success/failure prediction to provide an enhanced calendar-based user interface.

[0079] At step 104, the SCM system 14, via a retailer user interface in many cases, displays a calendar with available dates as shown in FIG. 8A. As illustrated in FIG. 8A, the calendar can show both available dates and unavailable dates as well as a time window or multiple time window options (not shown). Returning to FIG. 7, at step 106 the retailer user interface or SCM system 14 may detect the selection of one or more dates. It can be appreciated that the calendar interface shown in FIG. 8A is only one example and other interactive calendars can be used, for example, to enable a user to explore multiple options at the same time by selecting multiple days and/or multiple time windows within the same day (not shown). The SCM system 14 (e.g., via a retailer user interface) can display such options at step 108, and include not only the routes and time windows for the options, but also pricing and other data that depend on the carriers and modes used for that particular option. At step 1 10 the SCM system 14 detects selection of an option and continues with the process as described above, e.g., by executing steps 56 and 58. [0080] Turning now to FIG. 8B, an example of delivery option user interface 200 is shown. In the user interface 200, a number of options 202a, 202b, 202c are shown by way of example, with detail provided for only Option 1 , denoted by numeral 202a, for clarity of illustration. For each delivery option on a selected day or days, the SCM system 14 can include a price indicator 204 that indicates the price for that particular option 202. The user interface 200 can also include a date indicator 206 and a time window indicator 208 to indicate the parameters of this particular option 202. By accessing and utilizing the delivery prediction module 15, the SCM system 14 can also provide delivery prediction information 210. In this example, the legs or segments 212 of the delivery route are graphically mapped out, with a success indicator 214 provided for each segment 212. It can be appreciated that in this example, the segments 212 are assumed to either use different carriers and/or modes of transportation or be associated with different days and therefore have different parameters and likelihoods of being on time according to the time window indicator 208. It can also be appreciated that the size of the time window can be adjusted to provide better prediction results. For example, if a particular carrier or segment 212 of the route routinely misses a 2 hour time window, the SCM system 14 can adjust the time window indicator 208 to increase the time window to 3 hours thus adjusting the delivery prediction results. In this case, while the delivery prediction numbers look better, the time window may be less attractive than in another option 202 with similar delivery prediction numbers but a tighter time window.

[0081] The user interface 200 can also include a damage report(s) option 216 which allows the customer to browse and access damage reports for the carriers or vendors if applicable. The user interface 200 can also include a survey result(s) option 218 to augment the delivery prediction data 210 with third party information such as past customer reviews that can assist a customer in identifying a best option. For example, a certain carrier may have great statistics for being on time for their leg or overall route, but have a relatively higher number of damage claims suggesting that for at least some goods, this would not be the ideal option. By presenting multiple options 202 side-by-side as shown in FIG. 8B, the customer can have all of the useful information together to enable informed decisions to be made.

[0082] An example of operations performed by the prediction engine 48, machine learning module 50 to generate and use the data models 46 to prepare a delivery prediction will now be described, making reference to FIGS. 9 through 16D.

I . INTRODUCTION [0083] It is found that service failures are pervasive in supply-chain networks, with important consequences on their cost-efficiency and customer experience. The delivery prediction module 15 can be used for predicting and explaining the cause of such failures, focusing on the last-mile pickup and delivery of items at customer locations. Such services can be planned by optimizers solving some variations of the Vehicle-Routing Problem, in the present case the Pickup and Delivery Problem with Time Windows (PDPTW [1]). Solutions include routes, eventually associated to a specific truck and driver, defined as a sequence of stops associated with customer locations, where services are delivered, with time-window, capacity and precedence constraints. Routes start and end at dispatch centers, where trucks are loaded.

[0084] The module 15 aims to predict service failures that occur after the route was planned, such as customer not at home, services rescheduled by dispatch center, and service refused or canceled by customer. Such predictions could be leveraged by optimizers to find routes with reduced amounts of failures, for instance by adjusting the slack time to maximize the probability that customers will be at home at the time of the service. They could also be used directly by human planners, for instance to increase the amount of confirmation calls to customers in case of high failure probability.

[0085] Current approaches to solve the pick-up and delivery problem with time windows include heuristics [2], meta-heuristics [3] and exact methods [1] It is found that these solutions may not take service failures into account. Instead, the solutions suggested in the literature to reduce service failure in last-mile problems include Collection Delivery Points [4], reception boxes, and delivery boxes [5] While these methods can improve service efficiency, they also reduce customer satisfaction by not delivering items directly to their homes. Instead, the delivery prediction module 15 aims at (1) predicting failure probabilities, (2) identifying factors that predict failures, and (3) suggesting countermeasures to avoid failures.

[0086] One can break down the problem of failure prediction into multiple, independent supervised classification problems. Given a failure type, a goal is to predict if a stop will fail with this type. One can also look for associations between stop features and failure, to provide specific ranges of values that lead to increased or reduced failure rates.

[0087] The delivery prediction module 15 can build a classifier using Random Forests [6], as they are one of the most successful and frequently used methods for supervised classification. Random Forests belong to ensemble learning methods: they combine the predictions of multiple decision trees built from randomly-selected features, to reduce overfitting. Random Forests also provide measures of feature importance, which helps interpret classification results. To provide further insights on the failure causes, the delivery prediction module 15 can extract Association Rules between categorized feature values and specific failure types.

[0088] For this purpose, a large dataset of more than 500,000 pickup and delivery services that occurred in Canada over a period of 6 months has been analyzed, among which a few percentages failed. The dataset is strongly imbalanced toward successful services, as it is commonly the case in failure analysis. This is an issue for classifiers as they tend to focus on the majority class, leading to poor sensitivity. To address this issue, the analysis leveraged resampling methods to either undersample the majority class [7] or oversample the minority class [8]

[0089] The following has been provided:

[0090] - applying Random Forests to the prediction of stop failures in the pickup and delivery problem with time windows and precedence constraints;

[0091] - applying Association Rules to the identification of factors predicting failures types; and

[0092] - analyzing a dataset of 500,000 pickup and delivery services, on which the performance of the classifier is quantified and feature ranges highlighted, leading to increased failure probabilities, for 5 failure types.

[0093] Section II presents the dataset and methods used, and Section III details the results. After, there is a discussion on the performance of the Random Forest classifier, on the relevance of resampling methods, and on possible counter-measures to reduce failure rates.

I I. DATASET AND METHODS

A. Dataset

[0094] 1) Presentation: A dataset was extracted from the database of ClearDestination

Inc, a Montreal-based company providing services for supply-chain optimization. The dataset contains 523,643 pickup and delivery services grouped in 183,872 stops, scheduled between September 2017 and February 2018 in Canada. Tables 1 (a) and 1 (b) below, show the features describing the stops and their services.

Table 1 (b) - Features of Services

[0096] Features describe the customer location, position of the stop in the route produced by the optimizer, phone call status (phone calls are placed to the customer at specific times before the service), date, and service characteristics. Some features may be correlated: for instance, features describing geographical location are strongly

interdependent.

[0097] 2) Data representation: The dataset contains records where each stop is associated to one or more services, represented as a vector of variable size (3 services per stop on average). This is a problem since classifiers cannot work on spaces containing vectors of variable size. A solution could be to create one record for each service, and to replicate the stop features in all the services associated with the stop. However, such a replication would likely lead the classifier to overfit particular stops, and to rely, for instance, on the exact latitude and longitude of the stop to predict failures. This is not desirable in the present case, since it is an aim to predict failures in situations more general than particular stop locations that may never re-occur.

[0098] Instead, the services of a particular stop in a“master service” were aggregated, for which the value of categorical service features (Service Type, Retailer and Item

Manufacturer) was determined as the most frequent value among services in the stop, and the value of numerical features (Item Volume, Item Weight and Estimated Service Time) was the sum of the values among services in the stop. The status and failure type of the stop were set as the most frequent status and failure type among its services. The resulting aggregated dataset contains 183,872 stops with 6.68% of failures. FIG. 9 shows the failure distribution by failure type in the aggregated dataset.

[0099] 3) Failure types: focus on some of the most frequent failures, namely:

[00100] Customer not at home (NAH, frequency: 1.46%)

[00101] Stop rescheduled by dispatch center (SR, 0.80%)

[00102] Refused by customer (RC, 0.60%)

[00103] Canceled by customer (CC, 0.49%)

[00104] Not in Stock (NS, 0.34%)

[00105] Failure type Customer not at home (NAH), happens when the customer is not present at the time of the service. Failure type Stop rescheduled by dispatch center (SR), may happen due to any unexpected event in the supply chain, for instance construction in the delivery area, or inbound delays at the dispatch center. Failure type Refused by customer (RC), means that the item was delivered to the customer’s place, but the customer refused it, perhaps because it did not match their expectations. Failure type Canceled by customer (CC), means that the service was canceled at the customer’s location, for instance because the customer did not have cash. Finally, failure type Not in stock (NS), means that the item was not in stock at the dispatch center on the day of delivery, which may happen when the information is unknown at the time of the planning.

[00106] The failure types present in the dataset may not always be very accurately used. This is due to the fact that they are set by different actors in the supply chain, among which the planning company, the dispatcher and the drivers, who may interpret the failure types differently. In particular, some failure types may overlap, which may disturb the classification. [00107] 4) Pre-processing: the dataset was pre-processed to remove duplicate entries, to replace missing values with a constant (of -100), and to encode categorical features as numerical values. Only 3 features had missing values: Item Manufacturer (S5, missing rate of 90.8 %), Apartment number (C5, 82.7%), and Door number (C3, 8.6%). In the remainder, results involving Item Manufacturer and to some extend Door number should be interpreted carefully due to the high missing rate. Missing apartment numbers, however, make sense as many addresses simply do not include an apartment number.

[00108] 5) Dataset imbalance: the following 3 methods were used to deal with dataset imbalance. First, we oversampled the minority class using Synthetic Minority Oversampling Technique (SMOTE [8]). SMOTE generates synthetic entries in the minority class, here the class of failed stops, using random linear combinations of existing failed stops. The methodology applied SMOTE-regular using 2 nearest neighbors, and an oversampling ratio equal to the ratio between the number of elements in the majority class and the number of elements in the minority class (so that the resulting dataset is evenly distributed among both classes).

[00109] The methodology also evaluated NearMiss, a method that undersamples the majority class using k-nearest neighbors [7] NearMiss-3 was used which, for every element in the minority class, determines its nearest neighbors and keeps only the furthest ones. This method was applied with 3 nearest neighbors.

[00110] Finally, the methodology assessed a straightforward random undersampling of the majority class, as we suspected that SMOTE and NearMiss would be disturbed by their use of the Euclidean distance to determine nearest neighbors, which is questionable in the dataset.

B. Classification using Random Forest

[00111] The methodology trained Random Forest classifiers on the aggregated dataset, using a training ratio of 80% and 5-fold cross validation. Random Forests were used as implemented in scikit-learn version 0.19.1.

[00112] To find suitable values of the Random Forest parameters, a grid search was performed on a training dataset with different numbers of estimators, maximum depths of a tree, split criteria and minimum numbers of samples to split on a node. Table 2 below shows the selected parameter values.

Table 2 - Parameter Settings Used in Random Forest

[00113] The methodology also captured feature importance in the Random Forest classification, as reported by scikit-learn. It is defined by the scikit-learn developersl , as the total decrease in node impurity (weighted by the probability of reaching that node, which is approximated by the proportion of samples reaching that node) averaged over all trees of the ensemble. However, feature importance provides limited insights to interpret classification results. One of the reasons is that correlation among a group features reduces the mean importance in this group of features, as explained in [9] To further interpret the results, the methodology also extracted Association Rules as explained hereafter.

C. Association Rules

[00114] Association Rules were extracted from the aggregated dataset, to check the consistency of classification results and to provide further insights on their interpretation. To do so, the methodology categorized numerical features into deciles, and represented stops with vectors containing (1) such categorized features, (2) the initial categorical features, and (3) a binary feature representing the stop status (success or failure type). An Association Rule, written as“ antecedent ® consequent" , consists of two tuples, an antecedent and a consequent [10]

[00115] The methodology focuses on the rules where the consequent is a singleton containing a stop status. To represent features in the antecedent, postfix numerical features with _Dx, to indicate that the value is in the x ^th decile and categorical features with _Vx, to indicate that the value is x. A hypothetical example of rule is:

StartSlack_D3, Day_V4 ® Fail_NAH

[00116] Which measures the association between failure type“Customer not at home”, and stops where Start Slack is in the third decile and Day has value 4. It should be noted that Association Rules provide a measure of co-occurrence rather than causality.

[00117] To measure the relevance of a rule x ® y, define its interest ratio (IR) by comparing the frequencies of tuple x in the complete dataset C versus in the set F of failed stops (the stops that contain y):

[00118] Where Supdx), ^the support of tuple x in set S, is the number of occurrences of x in S. Focus on the cases where Supdx) ¹ 0, which gives 0 ¹ 0. Then define the interest ratio as follows:

[00119] The interest ratio measures the effect of x on the probability to fail with type y. Another way to understand it is to express its relation to the failure probability conditional to the presence of:

sup f ix)

x: P(y |x) =

Supcix )

[00120] Which gives:

P(y |x) = lR(x ® y)P{y) (0 ³ 0)

[00121] Or:

P(y|x) = IR(x®y ) (0 < 0)

[00122] Also compute the confidence of rule x ® y with the usual definition:

[00123] The following relation should finally be noted:

\F\

confix ® y) = 0

| C|

[00124] The methodology looks for rules with high interest ratio (large or small ø), and high frequency in the failed set (Sup_F x) ³ s, where s is the desired support threshold in F). One can find them using the following approach:

[00125] 1) Find the set / of items x in F s.t. Sup_F{x) ³ s.

[00126] 2) For every x in /, compute Siip_c(x).

[00127] Perform step 1) using the FP-growth algorithm [1 1 ], as implemented in Apache Spark version 2.3.1. Note that finding the frequent item sets in F requires much less memory than in C since |F| « | C| . Implement step 2) using a single pass on C, which does not raise any memory issue since |/| is small. To obtain a limited set of rules, we then select the Association Rules x ® y such that x e l and:

[00129] size(x ® y) is the size of the rule, i.e., the number of elements in x. R_t is the set of rules of size i and <_AIR is a partial order on R

defined as follows:

Vr₁ r₂ E R₁ x R₂, r = (x_t ® y), r₂ = (x₂, x₃ ® y)

I II . RESU LTS

A. Classification results

[00130] FIGS. 10A-10F shows the sensitivity (ratio of true positives) and specificity (1 - ratio of false positives) obtained for the different failure types and resampling methods. Without resampling, the sensitivity to failure remains 0, as expected in such an imbalanced dataset. Oversampling with SMOTE improves the sensitivity to an average of 0.36 while maintaining a high specificity of 0.92. Undersampling with NearMiss and Random

Undersampling further increases the sensitivity to an average of about 0.7, with a specificity close to 0.7. This appears to be the best compromise between sensitivity and specificity. On average, Random Undersampling performs slightly better than NearMiss. In the remainder, we focus on results obtained with Random Undersampling.

[00131] The classification performance is quite stable across failure types. With Random Undersampling, the best specificity values are obtained for NAH and NS, while RC is slightly under average. Sensivitiy values are close to average for all failure types, NS being slightly above.

B. Important features and Association Rules

[00132] 1) Customer not at home (NAH): FIG. 1 1A shows the feature importance resulting from the classification of failures of type“Customer not at home”. Feature labels refer to the ones in Tables 1 (a) and 1 (b), and feature importance is computed as explained in Section ll-B. Feature importance is largely dominated by a single feature,

Detailed_Call_Status (P3), peaking at an importance of 0.27. The next 18 features have similar importance values, ranging from 0.026 to 0.05. The remaining 13 features are between 0.00 and 0.022. [00133] FIG. 1 1 B shows the antecedents of the Association Rules with consequent FAIL_NAH, selected as described in Section ll-C, with their confidence and interest ratio (IR). For clarity, the elements of rules of size 1 are omitted in rules of size 2. For instance, Rule 1 means that (Detailed_Call_Status_V3 => FAIL_NAH) has a confidence of 0.036 and an interest ratio of 2.45, and Rule 2 means that (Detailed_Call_Status_V3,

Estimated_Service_Time_D2 => FAIL_NAH) has a confidence of 0.06 and an interest ratio of 4.07. Rules with f ³ 1 are represented in red, and rules with f < 1 are in green. For instance, Rule 1 , shown in red, means that Detailed_Call_Status = 3 increases failure probability by a factor of 2.45, while Rule 13, shown in green, means that

Detailed_Call_Status = 2 decreases failure probability by a factor of 2.06.

[00134] The rules in FIG. 1 1 B are consistent with the features importance in FIG. 1 1A, Detailed_Call_Status (P3) being the most important feature. They show that P3=3, which means that a call landed on voicemail, increases the failure probability by 2.45 times (Rule 1). This ratio increases to 3.67 if the call was marked failed (Rule 3: P2=5), to 4.07 if the estimated service time is shorter than 8 minutes (Rule 2: S6 in D2 (240-480]), or to 3.12 if the item volume is low (Rule 5: S3 in D1 [0.0, 0.002]), perhaps because less voluminous items are cheaper on average and customers give less value to them. The failure probability also increases if the item is delivered by specific companies (Rule 4: S2=3 and Rule 8: S2=39), in the Montreal/Laval area (Rule 6: C9=22), on a Tuesday, Wednesday or Thursday (Rule 9: D3=2, Rule 1 1 : D3=3, Rule 7: D3=4), if the time window starts between 6am and 8am (Rule 10: R7 in D1 (359.99, 480.0]), or if the service is planned between 10am and 12pm (Rule 12: D2=2).

[00135] Conversely, P3=2, which means that a call was answered by a human, reduces the failure probability by 2.06 times (Rule 13). Failure probability is further reduced if the item is lighter than 2 lbs (Rule 14: S4 in D1 [0.0, 2.0]), perhaps because drivers can leave small items unattended at the customer’s door when they agreed during a phone call. Finally, failure probability is also reduced if the address has no apartment number (Rule 15: C5 in D1) or if the time window provided to the customer is short (Rule 16: R9 in D2 (120.0,

180.0]).

[00136] 2) Stop rescheduled (SR): FIG. 12A shows the feature importance resulting from the classification of failures of type“Stop rescheduled”. The failure importance is more uniformly distributed than for NAH. Four features stand out: S2 (Retailer), D1 (Week of Year), R9 (Time_Window_Size) and R2 (Driver). Feature importance remains quite constant for the next 8 features, and it seems to decrease linearly to 0 for the remaining features. [00137] The Association Rules in FIG. 12B confirm the importance of the Retailer: some companies increase the failure rate (Rule 1 : S2=39, Rule 47: S2=149), and other ones reduce it (Rule 13: S2=3). The failure rate is also increased by high start slack times (Rule 9: R10 in D9 (120.0, 129.0], Rule 40: R10 in D8 (1 15.0, 120.0]), and by high end slack times (Rule 14: R1 1 in D6 (1 16.0, 120.0], Rule 44: R1 1 in D7 (120.0, 128.0]). The time window size also has an effect on the failure rate: D3 (180.0, 240.0] increases the failure rate (Rule 17, 46, 49, 54 and 58), while D2 (120.0, 180.0] reduces it (Rule 29). As for NAH, a call landing on voicemail (P3=3) increases the failure probability (Rule 34). Interestingly, services executed toward the end of the route tend to be rescheduled more often (Rule 55: R3 in D10 (16.0, 36.0]), and so do services with a short estimated job time (Rule 56: S6 in D1 (0.99, 240.0]). Finally, failures are also more frequent for services with a median weight (Rule 32: S4 in D5) or for volumes lower than 18.3 cf (Rule 42: S3 in D3 (1.45, 18.3], Rule 52: S3 in D2 (0.002, 1 .45]).

[00138] 3) Refused by customer (RC): FIG. 13A shows the feature importance resulting from the classification of failures of type“Refused by customer”. Feature S2, Retailer, is standing out again. The importance seems to decrease linearly for the next features, with a slight increase for C2 (Latitude) and S4 (ltem_Weight), and a slight drop between R3 and P3.

[00139] The Association Rules in FIG. 13B show the effect of R9 (Time_Window_Size): when in D3 (180.0, 240.0] (Rule 1), it reduces the failure rate by a factor of 1.78, while when in D4 (240.0, 300.0] (Rule 5), it increases it by a factor of 1.74. The failure rate also increases for company 8 (Rule 7: S2=8), for an estimated service time in D3 (480.0, 720.0] (Rule 1 1), in the Toronto area (Rule 16: C1 in D4 (-79.469, -79.254], Rule 18: C2 in D3 (43.595, 43.737], Rule 30: C9=23), for the highest start slack times (Rule 19: R10 in D10 (129.0, 751.0]), for services scheduled between 1 1 :39am and 12:08pm (Rule 24: R7 in D6 (700.0, 728.0]), and for voluminous items (Rule 28: S3 in D6 (49.02, 59.6]).

[00140] 4) Canceled by customer (CC): FIG. 14A shows the feature importance resulting from the classification of failures of type“Canceled by customer”. As in the two previous failure types, Retailer (S2) is standing out, and the importance seems to decrease linearly among the other features.

[00141] The Association Rules in FIG. 14B show the importance of Time_Window_Size (R9), as in the previous failure type: services tend to fail less when R9 is in D3 (180.0,

240.0] (Rule 1), and they fail more when R9 is in D2 (120.0, 180.0] (Rule 8, 12, 24, 36, 50 and 55) or in D4 (240.0, 300.0] (Rule 19). Two particular companies also have increased failure rates (Rule 13: S2=8, Rule 25: S2=3). The failure rate is also increased in the Montreal area (Rule 5: C6=21 18, Rule 5: C2 in D7 (45.452, 45.52], Rule 72: C2 in D8 (45.52, 45.619], Rule 10: C1 in D8 (-73.586, -73.289], Rule 22: C1 in D7 (-73.754, -73.586], Rule 33: C9=22), when a call lands on voicemail (Rule 45: P3=3), when the service is toward the end of the route (Rule 64: R3 in D10 (16.0, 36.0]), when the service time window starts around mid-day (Rule 69: R7 in D6 (700.0, 728.0]) or ends between 4pm and 5pm (Rule 70: R8 in D9 (960.0, 1020.0]), and when the item has a very low or close-to-average volume (Rule 51 : S3 in D1 [0.0, 0.002], Rule 59: S3 in D6 (49.02, 59.6]).

[00142] 5) Not in stock (NS): FIG. 15A shows the feature importance resulting from the classification of failures of type“Not in stock”. The most important feature is the week of the year (D1), followed by features related to geographical location (C2, C1 , C8 and C9), features related to the route (R9, R1 1 and R10), the company (S2) and the volume (S3).

[00143] This is consistent with the Association Rules in FIG. 15B. Note that the methodology used different filtering parameters for this failure type, due to the important number of rules with high IR. Rule 2 has an extremely high IR of 1 10.43, for a confidence of 37.4%: it means that company 158 had a not-in-stock failure rate of 37.4% in province 1 (Quebec). Some geographical locations spanning from Gatineau to Sorel-Tracy have increased failure rates (Rule 8, 42 and 51 : C1 in D7 (-73.754, -73.586], Rule 12 and 55: C1 in D6 (-75.601 , -73.754], Rule 9: C2 in D8 (45.52, 45.619], Rule 14: C2 in D9 (45.619, 46.328], Rule 60: C2 in D7 (45.452, 45.52]) while other ones have decreased failure rates (Ontario, Rule 49: C7=2). Two specific weeks have increased failure rates: week 36 (Rule 19), which was the week of Labor Day in 2017, and week 44 (Rule 50), which was the week of Halloween. As for route features, time windows shorter than 2 hours (Rule 5: R9 in D1 [0.0, 120.0]), negative end slack times (Rule 32: R1 1 in D1 (-537.001 , 62.0]), and start slack times between 53 and 63 minutes (Rule 21 : R10 in D3 (53.0, 63.0]) have increased failure rates, while time windows between 2 and 3 hours (Rule 72: R9 in D3 (180.0, 240.0]) reduce the failure rate. Specific companies increase the failure rate (Rule 1 : S2=158, Rule 41 :

S2=7) while other ones reduce it (Rule 71 : S2=3).

IV. DISCUSSION

A. Classification results

[00144] Random Forests showed good performance when applied to the dataset pre- processed with Random Undersampling: they reach an average sensitivity of 0.7 and an average specificity of 0.7. Thus, 70% of the failures of the studied types could be predicted, which represents 4,750 failed stops per year in the studied dataset. This prediction ability is an opportunity to save on pickup and delivery costs. The classification performance could be further improved by (1) improving the quality of the dataset, in particular through a better definition and separation of failure types, (2) improving the dataset aggregation technique to deal with records of non-uniform sizes; the technique essentially averages the features of the services in a stop, which leads to information loss, (3) improving the strategy to deal with dataset imbalance, perhaps through a more specific oversampling method. Regarding point (3), the poor performance of SMOTE compared to the other resampling methods is illustrated in FIGS. 16A-16D. The generated linear combinations of services may not be realistic. In particular, the generate services do not respect natural boundaries such as lakes or uninhabited regions, not to mention roads or actual addresses. This behavior is not surprising, since no such constraints were included in the oversampling method. Similar inconsistencies are also very likely to happen in other features. On the contrary, NearMiss and Random Undersampling maintain a realistic distribution of services, at the cost of reducing the dataset. A more constrained oversampling technique might be able to address this limitation.

B. Important features and Association Rules

[00145] Overall, the methodology observes a good agreement between the feature importance obtained from Random Forest and the selected Association Rules.

Nevertheless, most Association Rules can have a low confidence value, below 5%, which shows that failures are predicted from combinations of features rather than straightforward associations. One can conclude that Association Rules, computed and selected using the methods presented, are a relevant addition to RF feature importance to provide finer- grained interpretation.

[00146] It should be noted that the extraction of Association Rules focused on rules where the antecedent occurs more than s times among the failed stops. This explains why the selection was biased towards rules with f > 1 (rules displayed in red).

C. Suggested Countermeasures

[00147] The Retailer has a measurable effect on all the failure types. Specific investigations among the companies with failure rates higher than average can be conducted, to better understand the failure causes.

[00148] Failures of type“Customer not at home” (NAH) are fou nd to be very dependent on confirmation calls. In case such calls are not answered, additional ones should be scheduled, in particular if the estimated service time is short, if the item volume is low, if the item is not delivered on a Monday or Friday, if the time window starts before 8am, or if the service is planned between 10am and 12pm. In addition to these indicators, the trained Random Forest model could be used to recommend additional calls specifically for services predicted to fail. There might even be situations were multiple unanswered calls result in the service to be removed from the route, if the specificity could be made close enough to 1. These operations can be configured into the delivery prediction module 15 used by the SMC system 14.

[00149] Failures of type‘Stop rescheduled” (SR) are associated with many features related to the route (R3, R9, R10, R1 1) and a few other ones related to the type of service (S3, S4, S6). Such information could be included in optimizers, to facilitate the building of routes with less failures of this type. Start slack times longer than 2 hours lead to increased failure rates, which suggests that failures might happen due to delays in the route: dispatch centers might decide to skip stops when the service wont happen in the time window, which happens with higher probability when the start slack time is high. Likewise, services scheduled toward the end of the route are rescheduled more often than average, perhaps again due to delays in the route. The Time Window Size also has an effect on the failure rate: increased failure rates are observed for window size longer than 3 hours.

[00150] Failures of type‘Refused by customer” (RC) are also associated with route- related features (R7, R9 and R10), perhaps because delays lead impatient customers to refuse items. In addition, they seem to occur more frequently at specific geographical locations (Toronto area). Again, this information could be used by optimizers to build routes with less of such failures. Such zones might also be further investigated to understand the reasons for refused items. In addition, specific items (volume in D6 and estimated service time in D3) have an increased failure rate of this type, which might be reported to the manufacturers.

[00151] Failures of type“Canceled by customer” (CC) are also associated with route- related and geographical features (Montreal area), which could again be used by optimizers.

[00152] Finally, failures of type“Not in Stock” (NS) are strongly related to one specific retailer for which more than 10% of the services fail, and even 37% in Quebec. This should be reported to the retailer and further investigated.

[00153] For simplicity and clarity of illustration, where considered appropriate, reference numerals may be repeated among the figures to indicate corresponding or analogous elements. In addition, numerous specific details are set forth in order to provide a thorough understanding of the examples described herein. However, it will be understood by those of ordinary skill in the art that the examples described herein may be practiced without these specific details. In other instances, well-known methods, procedures and components have not been described in detail so as not to obscure the examples described herein. Also, the description is not to be considered as limiting the scope of the examples described herein.

[00154] It will be appreciated that the examples and corresponding diagrams used herein are for illustrative purposes only. Different configurations and terminology can be used without departing from the principles expressed herein. For instance, components and modules can be added, deleted, modified, or arranged with differing connections without departing from these principles.

[00155] It will also be appreciated that any module or component exemplified herein that executes instructions may include or otherwise have access to computer readable media such as storage media, computer storage media, or data storage devices (removable and/or non-removable) such as, for example, magnetic disks, optical disks, or tape. Computer storage media may include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules, or other data. Examples of computer storage media include RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by an application, module, or both. Any such computer storage media may be part of the SMC system 14 or delivery prediction module 15, any component of or related thereto, etc., or accessible or connectable thereto. Any application or module herein described may be implemented using computer readable/executable instructions that may be stored or otherwise held by such computer readable media.

[00156] The steps or operations in the flow charts and diagrams described herein are just for example. There may be many variations to these steps or operations without departing from the principles discussed above. For instance, the steps may be performed in a differing order, or steps may be added, deleted, or modified.

[00157] Although the above principles have been described with reference to certain specific examples, various modifications thereof will be apparent to those skilled in the art as outlined in the appended claims. REFERENCES

[00158] [1] S. Ropke and J.-F. Cordeau,“Branch and cut and price for the pickup and delivery problem with time windows,” Transportation Science, vol. 43, no. 3, pp. 267-286, 2009.

[00159] [2] S. Ropke and D. Pisinger,“An adaptive large neighborhood search heuristic for the pickup and delivery problem with time windows,” Transportation Science, vol. 40, no. 4, pp. 455-472, November 2006.

[00160] [3] W. P. Nanry and J. W. Barnes,“Solving the pickup and delivery problem with time windows using reactive tabu search,” Transportation Research Part B: Methodological, vol. 34, no. 2, pp. 107-121 , 2000.

[00161] [4] L. Song, T. Cherrett, F. McLeod, and W. Guan,“Addressing the last mile problem: transport impacts of collection and delivery points,” Transportation Research Record, vol. 2097, no. 1 , pp. 9-18, 2009.

[00162] [5] M. Punakivi, H. Yrjola , and J. Holmstrom,“Solving the last mile issue:

reception box or delivery box?” International Journal of Physical Distribution & Logistics Management, vol. 31 , no. 6, pp. 427-439, 2001.

[00163] [6] L. Breiman,“Random forests,” Machine learning, vol. 45, no. 1 , pp. 5-32,

2001.

[00164] [7] I. Mani and I. Zhang,“knn approach to unbalanced data distributions: a case study involving information extraction,” in Proceedings of workshop on learning from imbalanced datasets, vol. 126, 2003.

[00165] [8] N. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer,“Smote: synthetic minority over-sampling technique,” Journal of artificial intelligence research, vol.

16, pp. 321-357, 2002.

[00166] [9] R. Genuer, J.-M. Poggi, and C. Tuleau-Malot,“Variable selection using random forests,” Pattern Recognition Letters, vol. 31 , no. 14, pp. 2225-2236, 2010.

[00167] [10] R. Agrawal, R. Srikant et ai,“Fast algorithms for mining association rules,” in

Proc. 20th int. conf. very large data bases, VLDB, vol. 1215, 1994, pp. 487-499.

[00168] [1 1] J. Han, J. Pei, and Y. Yin,“Mining frequent patterns without candidate generation,” in ACM sigmod record, vol. 29, no. 2. ACM, 2000, pp. 1-12.

Claims

Claims:

1. A method of providing delivery options comprising:

interfacing a supply-chain management system with a retailer user interface;

detecting a request from the retailer user interface to schedule a delivery of one or more items through an integrated supply chain network coordinated by the supply-chain management system;

based on the request, determining from the integrated supply chain network, at least one delivery option for at least one delivery date, each delivery option comprising a route, one or more carriers for the route, and a time window;

for each delivery option, use a prediction engine and at least one data model to compute at least one delivery prediction parameter indicative of a likelihood of success for completing the route within the time window using the one or more carriers for the route, the at least one data model having been generated using historical delivery data; and

providing, in response to the request, each of the at least one delivery option augmented with the at least one delivery prediction parameter to enable the retailer user interface to display the at least one delivery option augmented with the at least one delivery prediction parameter in a user interface element enabling selection of a delivery date for the one or more items.

2. The method of claim 1 , wherein the at least one delivery prediction parameter comprises a percentage-based value indicative of the likelihood of success.

3. The method of claim 1 or claim 2, wherein the route comprises a plurality of segments, the prediction engine being applied to each of the segments to provide a likelihood of success value for each segment in the route.

4. The method of claim 3, wherein at least two of the segments comprise different carriers.

5. The method of claim 3 or claim 4, wherein at least two of the segments comprise different modes of transportation in an intermodal route.

6. The method of any one of claims 1 to 5, further comprising obtaining at least one damage report associated with a carrier to be used for the route in at least one delivery option, and providing damage report data with the associated delivery option.

7. The method of any one of claims 1 to 6, further comprising obtaining at least one survey result associated with a carrier to be used for the route in at least one delivery option, and providing survey result data with the associated delivery option.

8. The method of any one of claims 1 to 7, wherein the at least one data model has been generated by applying a supervised classification using a dataset of the historical data.

9. The method of claim 8, wherein the supervised classification comprises building a classifier using Random Forests to combine predictions of multiple decision trees built from randomly selected features in the dataset.

10. The method of claim 9, further comprising capturing feature importance values in the Random Forest classification.

1 1. The method of any one of claims 1 to 10, wherein prediction engine further extracts at least one association rule between categorized feature values and specific failure types to check consistency of classification results and to provide further insight into an interpretation of the classification results.

12. The method of any one of claims 1 to 1 1 , wherein the at least one data model is generated based on at least one of the following failure types: customer not at home, stop rescheduled, refused by customer, canceled by customer, and not in stock.

13. The method of any one of claims 1 to 12, further comprising:

providing a calendar user interface to the retailer user interface;

enabling selection of at least one delivery date from the calendar user interface; and when detecting a selection of a delivery date, displaying the at least one delivery option in an options user interface, wherein the options user interface graphically depicts the route augmented with the at least one delivery prediction parameter.

14. A computer readable medium comprising computer executable instructions for performing the method of any one of claims 1 to 13.

15. A system comprising a processer and memory, the memory comprising computer executable instructions for performing the method of any one of claims 1 to 13.