JP2020187416A - Physical distribution management system - Google Patents

Abstract

To provide a technique to support the overall optimization of physical distribution management.SOLUTION: A predictor has: an acquisition part for acquiring product information of e-commerce products; and a control part for generating a third behavior model based on the reinforcement learning of a first behavior model and a second behavior model to calculate mutually different rewards using the acquired product information and environmental state variables for multiple allowed behavior variables in an environment related to a physical distribution of the products, and predicting an optimal behavior variable in the environment using the third behavior model.SELECTED DRAWING: Figure 1

Description

The present invention relates to a physical distribution management system.

In logistics management in the supply chain, the so-called Bullwhip Effect (hereinafter "BE")
Is known). BE is a phenomenon in which demand propagates from the downstream to the upstream of the supply chain while the demand expands as a result of demand forecasting and decision making in the downstream of the supply chain. Since this phenomenon leads to excess inventory and shortages, the mechanism of occurrence and the method of suppressing BE have been the subject of research for many years. Factors that cause BE include price list, ordering frequency, return policy, frequency and depth of price sales measures, degree of information sharing, demand forecasting method, allocation rule when out of stock, and so on.

In addition, Beer Game (hereinafter referred to as "BG") is known as one method for verifying BE. BG is a simulation game in the supply chain of beer connected in series, and each player of 4 players (retailer, wholesaler, distributor, manufacturer) competes for cost minimization within a fixed period.

The process of BG is known as the Markov Decision Process (MDP), and in BG, the only information that each player can observe is the order and exchange of goods with adjacent players and their own inventory level, so the so-called Partially Observable. Markov Decision Process (POMDP). In BG, each player selects an action that minimizes costs from observable information. However, BG has a large observation space and action space, and handles a non-stationary time series, which poses a complicated problem. Therefore, it has been shown that by applying deep reinforcement learning to BG, it is possible to realize overall optimization of the distribution process in the supply chain (Non-Patent Documents 1-3).

V. Mnih et al. Human-level control through deep reinforcement learning, doi 10.1038 / nature14236 (2015). Afshin Oroojlooyjadid et al. A Deep Q-Network for the Beer Game: A Reinforcement Learning Algorithm to Solve Inventory Optimization Problems, arXiv: 1708.05924v2 (2018). Taiki Fuji et al. Deep Multi-Agent Reinforcement Learning using DNN-Weight Evolution to Optimize Supply Chain Performance. Doi 10.24251 / HICSS.2018.157 (2018).

In mail-order sales using the Internet (online mail order), the difference between the efficiency of information flow in cyberspace and the efficiency of goods flow in physical space is becoming noticeable, and this difference is a new cause of BE. There is a possibility of becoming. For example, Electronic Commerce (
As a result of increased fluctuations in demand due to highly efficient sales measures on EC) sites, delivery delays, shortages, excess inventory, etc. may occur. Therefore, even if the above-mentioned conventional technology is used, it may not be possible to realize the overall optimization of the distribution process in the supply chain for online shopping, and it has been established to realize such an overall optimization. No method was proposed.

Therefore, the technology disclosed in this case was made in view of the above circumstances, and the purpose thereof is to provide a technology that supports the overall optimization of physical distribution management.

The prediction device disclosed in this case is an acquisition unit that acquires product information of products in electronic commerce, and the acquired product information and the state of the environment for multiple values of behavior variables that are allowed in the environment related to the distribution of products. A third behavior model is generated based on the reinforcement learning of the first behavior model and the second behavior model that calculate different rewards using variables, and the optimum behavior in the environment is generated using the third behavior model. It has a control unit that predicts the value of a variable. As a result, this prediction device can generate a balanced behavior model that achieves overall optimization from two behavior models for which conflicting goals are set in product distribution management, and predict the optimum behavior variable. it can.

Further, in the above prediction device, the behavior variable is the number of ordered products by the seller of the product, and the product information acquired by the acquisition unit changes the number of ordered products, the number of shipments, and the number of arrivals of the products over a certain period. The information to be shown, the environmental state variables are the sales of the product, the purchase price of the product, the shortage cost related to the shortage of the product, the sales promotion cost related to the promotion of the product, and the excess inventory of the product. The related inventory cost and the delivery cost related to the delivery of the product, and the control unit uses the acquired product information and the environmental state variable for each of the number of orders corresponding to multiple values of the behavior variable. A third behavior model may be generated based on the strengthening learning of the first behavior model and the second behavior model that calculate different rewards, and the optimum number of orders may be predicted using the third behavior model. .. As a result, the BEI is controlled so as not to exceed the threshold value, and the shipment is performed so as to satisfy the number of demands as much as possible, thereby balancing the conflicting goals of minimizing inventory cost and maximizing sales. It is possible to optimize the inventory of products.

Further, in the above prediction device, the action variable is the human time related to the storage of the product in the warehouse, and the product information acquired by the acquisition department is the information indicating the implementation date of the sales promotion of the product, and is the environment. The state variable is the sales promotion cost determined according to the day of the sales promotion of the product, and the control unit uses the acquired product information and the acquired product information for each of the human hours corresponding to the plurality of values of the behavior variable. A third behavior model is generated based on the reinforcement learning of the first behavior model and the second behavior model that calculate different rewards using the state variables of the environment, and the third behavior model is used to be optimal. You may predict man-time. This balances the conflicting goals of maximizing sales and minimizing shipping costs by shipping to meet the number of demands that fluctuates due to sales promotion and keeping man-hours low in the warehouse where products are stored. It is possible to optimize the man-hours assigned to the shipment of the product on the sales promotion implementation date set in this simulation.

Further, in the above prediction device, the action variable is a value indicating the ease with which the product is recommended on the EC site, and the product information acquired by the acquisition unit is displayed as the number of clicks on the product over a certain period on the EC site. It is information indicating the number of times, and the state variables of the environment are the size of the product, the sales of the product, the purchase price of the product, and the inventory cost determined according to the size of the product. The first behavior model and the second behavior that calculate different rewards using the acquired product information and the environmental state variables for each value indicating the ease of product recommendation corresponding to multiple values. A third behavior model may be generated based on the strengthening learning of the model, and the third behavior model may be used to predict the optimum value indicating the ease with which the product is recommended. By doing this, it is possible to determine which products cannot be sold because they are not recommended and which products cannot be sold even if they are recommended, and to recommend products that are more likely to sell, thereby optimizing product recommendations on the EC site. Can be planned.

Further, in the above prediction device, the action variable is an incentive given to the user by changing the designated delivery date of the product on the EC site, and the product information acquired by the acquisition unit is the delivery destination and delivery of the product. Information indicating the date and time, the delivery area of the delivery company of the product, the number of delivery destinations that can be delivered, the time required for delivery of the product, and the delivery distance in the delivery of the product, and the control unit controls multiple values of the action variable. A third behavior model based on the strengthening learning of the first behavior model and the second behavior model that calculates different rewards using the acquired product information and the environmental state variables for each of the incentives corresponding to And a third behavioral model may be used to predict optimal incentives. In this way, while balancing the two goals of minimizing delivery costs and minimizing incentive costs, the size of the incentive given when prompting the user to change the delivery date of the product on the EC site is optimized. be able to.

According to the technology disclosed in the present case, it is possible to provide a technology that supports the overall optimization of physical distribution management.

FIG. 1 shows an example of a prediction device according to the first embodiment. FIG. 2A shows the relationship between the player and the supply chain in BG, and FIG. 2B shows the relationship between the player and the supply chain in the Netshop Game executed in the first embodiment. FIG. 3 shows an example of a simulation algorithm executed in the first embodiment. FIG. 4 shows an example of the environmental parameters set in the first embodiment. FIG. 5 shows an example of the Netshop Game reward set in the first embodiment. FIG. 6 shows an example of updating each of the above state variables executed in the first embodiment. FIG. 7 shows an example of a goal condition for a meta player in the first embodiment. FIG. 8 is a graph showing an example of changes in time steps and rewards for a meta player in the first embodiment. FIG. 9 is a graph showing an example of the transition of the acquisition reward of the cyber player in the first embodiment. FIG. 10 is a graph showing an example of a transition of the acquisition reward of the physical player in the first embodiment. FIG. 11 is a graph showing an example of the transition of the acquisition reward of the meta player in the first embodiment. FIG. 12 shows an example of the simulation result of each player in the first embodiment. FIG. 13A shows an example of the simulation reward in the modified example 1, FIG. 13B shows an example of updating the state variable of the simulation in the modified example 1, and FIG. 13C shows an example of the goal condition of the simulation in the modified example 1. Shown. FIG. 14A shows an example of the simulation reward in the modification 2, FIG. 14B shows an example of updating the state variable of the simulation in the modification 2, and FIG. 14C shows an example of the goal condition of the simulation in the modification 2. Shown. FIG. 15A shows an example of the simulation reward in the modified example 3, FIG. 15B shows an example of updating the state variable of the simulation in the modified example 3, and FIG. 15C shows an example of the goal condition of the simulation in the modified example 3. Shown.

Hereinafter, preferred embodiments of the techniques disclosed in the present disclosure will be described with reference to the drawings. However, the configuration of the components described below should be appropriately changed depending on the configuration of the device to which the technology disclosed in the present disclosure is applied and various conditions. Therefore, it is not intended to limit the technical scope of the technology disclosed in this case to the following description.

(First Embodiment)
FIG. 1 is a diagram showing a schematic configuration of a prediction device according to the first embodiment. As shown in FIG. 1, the prediction device 1 includes a control unit 11, a storage unit 12, an operation unit 13, a display unit 14, and a communication unit 15.

In online shopping, there are differences in characteristics between processes that are completed in cyberspace such as transactions via EC sites and the Internet, and processes that are performed in physical spaces such as distribution warehouses and distribution centers. For example, in cyberspace, 100 products are just numerical data. For this reason, it can be said that it is less likely to be physically restricted to increase the number of units sold by 10 times to 1000 pieces by bullish sales measures. On the other hand, in the physical space, 100 products are entities having volume and weight. Therefore, it can be said that increasing the number of units sold to 1000 is greatly affected by physical restrictions such as warehouse capacity and shipping capacity. Further, in the above BG, there is a lead time in an order to an upstream player in the supply chain, but the lead time of the order can be ignored in a transaction in online shopping. Therefore, when dealing with the problem of the distribution process of online shopping, it may not be possible to find a solution to the problem even if the above BG model is used.

Here, FIG. 2 shows the relationship between the player and the supply chain in each of the BG (FIG. 2A) and the Netshop Game (FIG. 2B) executed in the first embodiment. In the first embodiment, the prediction device 1 executes a simulation called Netshop Game in which the environment illustrated in FIG. 2A is set. As shown in FIG. 2B, in the above BG, four players, a retailer, a wholesaler, a distributor, and a manufacturer, are connected in series, but in Netshop Game, the players are only retailers, and the players are referred to the retailers. The three players, the wholesaler, the distributor, and the manufacturer, which are located on the upstream side of the supply chain, are combined into one player as a supplier. In the first embodiment, it is assumed that there is only one type of electronic commerce product handled by Netshop Game. In addition, Netshop Game divides retailers into cyber players who manage processes that occur in cyberspace and physical players who manage processes that occur in physical space.

The cyber player acts so that the fill rate (also referred to as FR), which indicates how well the product can be supplied in response to the demand from the customer, becomes larger than a preset threshold value. Cyber players aim to maximize rewards in cyberspace. Cyber players actively carry out sales promotions such as sales on EC sites, for example. Cyber players also ignore the costs of increased inventories and act to minimize lost opportunities due to shortages.

In addition, the physical player acts so that the Bullwhip Effect Index (also referred to as BEI) becomes smaller than a preset threshold value. Physical players aim to minimize logistics costs. Physical players ignore lost opportunities due to shortages and act to minimize inventory and keep BEs low in order to minimize costs associated with warehousing and delivery.

In the first embodiment, the environment shown in FIG. 2B is implemented in the prediction device 1 by OpenAI Gym, and the optimal actions for achieving the above objectives of the cyber player and the physical player are learned by Deep Q-Network (DQN). And evaluate. In addition, the first
In one embodiment, a meta player is introduced that acts to balance the rewards of the cyber player and the physical player and the goal conditions of the Netshop Game. It is an example of a first behavior model and a second behavior model in which a cyber player and a physical player calculate different rewards by using product information and environmental state variables. Further, the meta player is an example of a third behavior model generated based on reinforcement learning of the first behavior model and the second behavior model.

ここで、ＩＬ_ｔは、タイムステップｔにおける在庫数、ＯＯ_ｔは供給業者に対して発注済みであるが未入荷の状態である商品数、ｄ_ｔは顧客からの商品の需要数、ＲＳ_ｔは、供給業者から入荷した商品数、ＳＳ_ｔは、顧客に出荷済みの商品数、ａ_ｔは、タイムステップｔにおいて発生するアクション、すなわち供給業者への商品の発注数である。このように、第１実施形態では、これら６変数の回帰分析を用いる。 Next, the details of the settings for each player in Netshop Game will be described. First, in Netshop Game, defined by the following equation observable state variables o _t at time step t (1).

Here, IL _t is the number of products in stock at the time step t, OO _t is the number of products that have been ordered from the supplier but are not in stock, _dt is the number of products _requested by the customer, and RS _t is , items that arrived from the supplier, SS _t is, items shipped to the customer, a _t is, actions that occur in the time step t, that is, the order quantity of goods to suppliers. As described above, in the first embodiment, the regression analysis of these six variables is used.

Further, the Netshop Game, from game start to a goal conditions achieve the episode is stored as indicating the state variable ho _t of all time steps in one episode in equation (2) below.

Next, the action space in Netshop Game will be described. Actions can also be said to be permissible behavioral variables in the environment related to the distribution of goods. In addition, each value of the action becomes each value of the action variable. As shown in the above action, the action in this embodiment is the number of orders for goods from the supplier. If the action space, that is, the degree of freedom in the number of ordered products allowed by the player, that is, the range from the lower limit to the upper limit of the ordered number is too wide, the processing efficiency in the prediction device 1 may decrease. Therefore, in the present embodiment, as an example, the Netshop Game is implemented with the action space as a set of discrete values [0, 1, 2, ..., 20] from 0 to 20.

Next, the reward in Netshop Game will be explained. In the above BG, as shown in the following formula (3), the reward is determined by the number of stocks in the time step t.

ここで、変数ｘについて（ｘ）^＋：ｍａｘ（０，ｘ）、（ｘ）⁻：ｍａｘ（０，−ｘ）である。また、在庫数が正の場合は在庫数分の在庫コストｃｈを乗算し、在庫数が負の場合は欠品による機会損失コストｃｐを乗算する。また、ｉは１〜４までの整数であり各値
が各プレーヤーに対応する。したがって、式（３）によって全プレーヤーの報酬の総計が算出される。

Here, for the variable x, (x) ⁺ : max (0, x), (x) ⁻ : max (0, −x). If the number of stocks is positive, the stock cost ch for the number of stocks is multiplied, and if the number of stocks is negative, the opportunity loss cost cp due to shortage is multiplied. Further, i is an integer from 1 to 4, and each value corresponds to each player. Therefore, the total reward of all players is calculated by the formula (3).

ここで、ｓ_ｐは売値、ｃ_ｒは仕入れ値、ｃ_ｓは販売促進費、ｃ_ｄは配送費である。Netshop Gameでは各プレーヤーはこれらの値を観測できないものとする。 In Netshop Game, selling price, purchasing price, sales promotion cost, and shipping cost are added to the calculation of reward. Specifically, the cyber player reward is calculated by the formula (4), the physical player reward is calculated by the formula (5), and the meta player reward is calculated by the formula (6).

Here, _{s p} is the selling price, _{c r} is Shiirene, _{c s} sales promotion expenses, and _{c d} is the shipping costs. Netshop Game does not allow each player to observe these values.

As shown by the formula (4), in addition to the opportunity loss cost of the shortage, the cyber player adds the difference between the orders to the supplier to the reward as the promotion cost when the number of orders is larger than the demand. Further, as shown by the equation (5), the physical player adds the inventory cost for the number of stocks and the delivery cost for the number of shipments to the customer to the reward. The rewards are set so that cyber players do not care about excess inventory and physical players do not care about shortages, because it intentionally creates a situation where a local optimum solution is sought. Is. On the other hand, as shown by the equation (6), the metaplayer adds each of the above costs to the reward so as to obtain the overall optimal solution that balances the above orientations.

Next, the goal conditions of each player in Netshop Game will be described. In the above BG, each player competes based on the total value of rewards within a predetermined time step period, but in Netshop Game, the following two indicators are set as goal conditions. Each player ends one episode when he or she achieves any of the indicators.

ここで、ｄｅｍａｎｄは、タイムステップｔにおける顧客からの需要数の直近ｐ期間分の配列である（ｐは正の整数）。また、ｓｈｉｐｐｅｄは、タイムステップｔにおける商品の出荷数の直近ｐ期間分の配列である。また、Ｖａｒ（ｘ）は、変数ｘの分散、Ｍｅａｎ（ｘ）は、変数ｘの平均である。さらに、Netshop Gameでは、以下の式（９）および式（１０）によって各プレーヤーのゴール条件の達成を判断する。

ここで、式（９）がフィジカルプレーヤーのゴール条件に用いられ、式（１０）がサイバープレーヤーのゴール条件に用いられる。また、式（９）および式（１０）がメタプレーヤーのゴール条件に用いられる。 One index of the goal condition in Netshop Game is BEI represented by the following formula (7), and the other index is FR represented by the following formula (8).

Here, demand is an array for the latest p period of the number of demands from customers in the time step t (p is a positive integer). Further, skipped is an array for the latest p period of the number of goods shipped in the time step t. Var (x) is the variance of the variable x, and Mean (x) is the mean of the variable x. Further, in Netshop Game, the achievement of the goal condition of each player is judged by the following equations (9) and (10).

Here, equation (9) is used for the goal condition of the physical player, and equation (10) is used for the goal condition of the cyber player. Further, the equations (9) and (10) are used as the goal condition of the meta player.

なお、初期値は、１から１０までの自然数からランダムに選択された値が採用される。 Next, the number of customer demands generated by Netshop Game will be described. In Netshop Game, the number of demands Dt from customers in time step t is a random variable x that follows a normal distribution to the average value of the number of demands for the latest p period of time step t, as shown in the following equation (11). In addition it is generated.

As the initial value, a value randomly selected from a natural number from 1 to 10 is adopted.

FIG. 3 shows an example of a simulation algorithm executed by OpenAI Gym in this embodiment. As shown in Fig. 3, Netshop Game uses the so-called Epsilon Leady Algorithm as a method of balancing the accumulation and utilization of experience obtained from the results of actions, and performs state evaluation using DNN (Deep Neural Network). .. Since the algorithm of FIG. 3 is a well-known algorithm described in the above non-patent documents, detailed description thereof will be omitted here.

Next, the parameters of each player and the definition of the reward calculation in this embodiment will be described. FIG. 4 shows an example of the environmental parameters set by the OpenAI Gym. As shown in FIG. 4, in the simulation executed in the present embodiment, the BEI threshold value (“bei_threshold”) used for the goal condition is 0.95, and the FR threshold value (“fillrate_threshold”) is 0.
95, inventory cost ("stock_cost") is 0.5, shortage cost ("shortage_cost") is 1.
.. 0, sales promotion cost ("promotion_cost") is 0.01, delivery cost ("delivery_cost")
) Is 0.01. In addition, to generate the number of demands from customers, "demand_volatility" in the figure
(Demand fluctuation), "demand_min" (minimum value), and "demand_max" (maximum value) are used. The selling price of the product (“sales_price”) is 2.0, and the purchase price of the product (“purchase_price”) is 1.2.

FIG. 5 shows an example of Netshop Game rewards set by OpenAI Gym. As shown in FIG. 5, rewards for each player of the cyber player, the physical player, and the meta player are set.

To explain the reward of the meta player more specifically, if the product is out of stock, the shortage cost for the number of stocks is added to the meta player (“cost + = abs (IL) * self.” In the figure. shortage_cost if IL <0 else 0 "). If the number of orders is larger than the number of orders from customers, the sales promotion cost is added ("cost + = (action --d) * self.promotion_cost if action> d else 0" in the figure). In addition, the inventory cost for the number of inventories is added to the current number of inventories (“cost + = IL * self.stock_cost if IL> 0 else 0” in the figure). In addition, the delivery cost is added to the number of shipments (“cost + = SS * self.delivery_cost if SS> 0 else 0” in the figure). In addition, the selling price (sales) is the value obtained by multiplying the number of shipments by the selling unit price (“sales_price = SS * self.sales_price” in the figure). In addition, the purchase price is the value obtained by multiplying the number of arrivals by the purchase unit price (“purchase_price = RS * self.purchase_price” in the figure). And the reward is
It is the value obtained by dividing the purchase price and each of the above costs from the sales (“reward = abs (sales_price)-abs (purchase_price)-abs (cost)” in the figure). The meta player aims to achieve the goal condition described later based on the reward set in this way.

Next, FIG. 6 shows an example of updating each of the above state variables executed in OpenAI Gym. As shown in FIG. 6, in one time step, the current stock quantity (“IL = self.state [0]” in the figure) and the number of unstocked orders (“OO = self.state [1]” in the figure). ), The number of orders in the previous time step (“a = self.state [5]” in the figure) is used. In addition, by adding the number of fluctuations that follow a normal distribution to the moving average of demand, the number of demands from customers is generated using random numbers (in the figure, "d = max (self.demand_min, int (dave + np.random)". .randn () * self.demand_volatility)) "). In addition, it is assumed that the products corresponding to the number of orders are received with the lead time of the number of arrivals as 1 (“RS = a # received shipment = last order (shipping lead time = 1” in the figure).
) ").

Then, first, the number of products received from the supplier is added to the number of stocks (“IL + = RS” in the figure). After that, if the number of demands from the customer is smaller than the number of stocks (“if d <IL:” in the figure), the number of products corresponding to the number of demands is shipped to the customer (“SS = d” in the figure), and the customer If the number of demands for is greater than or equal to the number of stocks (“else: # d> = IL” in the figure), the number of stocks is set as the number of shipments to customers (“SS = IL” in the figure), and the number of stocks is changed (“SS = IL” in the figure). In the figure, "IL-= SS"). Then, the value obtained by dividing the number of orders received this time by the number of orders received is added to the number of orders not received (“OO + = action --RS” in the figure). In this way, each state variable is changed, and the changed state variable is used to order, receive, and ship the goods in the next time step.

FIG. 7 shows an example of a goal condition for a meta player. As shown in FIG. 7, the variance of the number of shipments (“vars4 = np.var (shipped)” in the figure) and the variance of the number of demands (“vars2” in the figure) in the most recent fixed period (for example, 100 days) from the current time step. = np.var (demand) ") and calculate the BEI from the calculated variance ("bei = vars4 / vars2 if vars2! = 0 else bei_threshold "in the figure). Then, when the calculated BEI is smaller than the above BEI threshold value (“if bei <bei_threshold:” in the figure), it is assumed that the calculated BEI satisfies the BEI condition (“bei_flag = True” in the figure). Also, the most recent fixed period from the current time step (
For example, the average number of shipments to the number of demands in 100 days) is calculated (“fill rate = np.mean (shipped / demand)” in the figure). Then, when the calculated average is larger than the above-mentioned FR threshold value (“if fillrate> self.fillrate_threshold:” in the figure), it is assumed that the calculated average satisfies the Filllate condition (“fill_flag = True” in the figure). .. Then, when both of the above two conditions are satisfied, it is considered that the goal condition has been achieved (“done = fill_flag & sum_flag” in the figure).

Each player repeats the update of the state variable shown in FIG. 6 at each time step with the aim of achieving the above goal condition based on the reward set as shown in FIG. FIG. 8 is a graph showing an example of changes in time steps and rewards for a meta player when reinforcement learning is performed with the upper limit of time steps set to 50,000 steps in Netshop Game. The horizontal axis of the graph represents the time step, and the vertical axis of the graph represents the reward that the meta player gets as described above. As shown in the graph of FIG. 8, it can be seen that the reward of the meta player increases toward a constant value as the time step progresses. Therefore, it can be said that the learning of the meta player is accumulated as the time step progresses.

9 to 11 show an example of changes in the number of demands, the number of orders, and the number of inventories in a test in which the behavior model of each player learned by the above Netshop Game is used and the Netshop Game is executed again for 100 episodes. It is a graph which shows. The horizontal axis of the graph represents the time step (“step” in the figure), and the vertical axis of the graph represents the number of products (“items” in the figure). Note that FIG. 9 shows the transition of the acquisition reward of the cyber player, FIG. 10 shows the transition of the acquisition reward of the physical player, and FIG. 11 shows the transition of the acquisition reward of the meta player. Here, it is assumed that the player has already learned is a state in which the player's learning has been accumulated to a state in which the reward acquired by the player changes to a substantially constant state, as shown in FIG. Further, the upper limit of the number of time steps in one episode is set to 1000 steps, and the episode ends when the player completes the action of the time step of the 1000th step without achieving the above goal condition. The graph in each figure shows the transition of the earned reward for the last 100 steps of each episode, that is, 100 steps back from the end of the episode. Further, in each figure, "stock" represents the number of stocks, "demand" represents the number of demands, and "shipped" represents the number of shipments.

As shown by the transition of the inventory quantity in FIG. 9, it is considered that the excess inventory tends to increase and remain high due to the behavior of the cyber player in Netshop Game. Further, in FIG. 10, since the state in which the number of stocks is 0 continues over a plurality of steps, it is considered that shortages tend to occur frequently due to the actions of the physical player. On the other hand, in FIG. 11, the graph showing the transition of the number of inventories is generally saw-shaped. It is considered that this is because the inventory quantity increases within the next step or within a few steps even if the inventory quantity becomes 0, and even if the inventory quantity increases too much, the demand quantity and the shipment quantity work to push down the inventory quantity. be able to. Therefore, it is possible that a balanced inventory can be achieved by the actions of the meta player.

FIG. 12 shows the results of calculating the average value of 100 episodes for the number of steps required for each player to achieve the game conditions, the BEI value, the FR value, and the earned reward value in the above test. In the graphs of FIGS. 9 to 11, the smaller the number of steps (“steps” in the figure) required to achieve the goal condition, the better. Further, when the value of BEI is smaller than 1.0, it is considered that the influence of BE is suppressed. In addition, the value of FR indicates the ratio of shipment to the demand from the customer, and the closer it is to 1.0, the better. As can be seen from FIG. 12, the cyber player is optimal if the purpose is only to maximize the reward, but as can be seen from the transition of the inventory quantity shown in FIG. 9, in the case of the cyber player, the inventory quantity is large in the episode. The phenomenon that the state continues is occurring. One of the reasons why this phenomenon occurs is that the setting of rewards for cyber players does not give an element to suppress excess inventory.

As shown in FIG. 12, in the case of the physical player, the inventory quantity is maintained at a small value, and therefore the BEI is also a lower value than that of the cyber player, but as can be seen from the transition of the inventory quantity shown in FIG. The state where the number of stocks is 0 may continue over a plurality of steps, that is, the state of shortage may continue, and as a result, the value of FR is also low.

Further, as shown in FIG. 12, in the case of the meta player, the earned reward is lower than that of the cyber player and the physical player, but the BEI value is smaller than that of the cyber player and the physical player, and the FR value is The value is between the case of a cyber player with more excess inventory and the case of a physical player with more shortages. Further, in the graph showing the transition of the inventory quantity in FIG. 11, the inventory quantity, the arrival quantity, and the shipment quantity are well-balanced because there is a part in which the cycle and the fluctuation range of the inventory quantity change are substantially constant over a plurality of steps. That is, it is considered that the learning effect of proper inventory quantity management for stabilizing the inventory is obtained.

In the first embodiment, the control unit 11 of the prediction device 1 controls the communication unit 15 as the acquisition unit to communicate with the product information DB (database) 2. The communication unit 15 acquires information on the number of orders, the number of shipments, and the number of arrivals of products in a certain period, which is stored in the product information DB 2. The information acquired from the product information DB 2 is an example of product information indicating fluctuations in the number of orders, the number of shipments, and the number of arrivals of products over a certain period. The control unit 11 generates the above behavior model by reinforcement learning that calculates a reward determined based on each state variable based on the information acquired by the communication unit 15 while changing the number of orders. Then, the control unit 11 uses this behavior model to simulate the behavior of the meta player in the Netshop Game.

Specifically, the control unit 11 repeats the simulation while changing the number of ordered products by the meta player from the upper limit to the lower limit, for example, increasing the number of orders by 1 from 0 to 20 in the above example. As an example, the control unit 11 sets the upper limit of the number of time steps of one episode to 1000 steps as described above, and when the meta player completes the action of the 1000th step time step without achieving the above goal condition, the episode Based on the result of executing 100 episodes for each order quantity, the average value of 100 episodes for the number of steps to achieve the goal condition for each order quantity, BEI value, FR value, and acquisition reward is calculated. The calculation result is output by calculating and storing the calculation result in the storage unit 12, displaying it on the display unit 14, and transmitting it from the communication unit 15 to an external device (not shown). The calculation result by the prediction device 1 is an example of the optimum number of orders predicted by the prediction device.

The user can confirm the calculation result by the prediction device 1 and determine the actual number of orders for the product targeted for the simulation. Therefore, according to the first embodiment, the number of ordered products by the seller is adjusted based on the calculation result by the simulation by the prediction device 1. As a result, the BEI is controlled so as not to exceed the threshold value, and the shipment is performed so as to satisfy the number of demands as much as possible, thereby balancing the conflicting goals of minimizing inventory cost and maximizing sales. It is possible to optimize the inventory of products.

The above is the description of the present embodiment, but the configuration and processing of the prediction device 1 of the present embodiment are not limited to the contents of the above description, and are within the range that does not lose the identity with the technical idea of the present invention. Various changes can be made in. A modified example of the above embodiment will be described below. In the following description, the same reference numerals will be given to the same configurations and processes as described above, and detailed description thereof will be omitted.

(Modification example 1)
In the first modification, a warehouse worker who stores an inventory of products in line with a sales promotion strategy for one product sold on an EC site is determined as a man-hour required for daily shipment. In the simulation executed in this modification, it is assumed that one time step is one day, the day of the week is set for each day, and the day when the product sales promotion is carried out is set in advance. Examples of sales promotion implementation dates include days including 5 (5th, 15th, 25th) and 50th. The information indicating the sales promotion implementation date may be stored in the product information DB 2, for example, and may be acquired by the prediction device 1 from the product information DB 2, or may be input by the user by operating the operation unit 13. .. FIG. 13 shows an example of the reward of this simulation set by the OpenAI Gym in the prediction device 1 (FIG. 13A) and an example of updating each state variable executed in the OpenAI Gym (FIG. 13B). The action (behavior variable) in this simulation is the man-hour required for shipping the product in one day, and a value from 0 to a predetermined upper limit is selected.

As shown in FIG. 13A, in the simulation executed in this modification, the absolute value of the cumulative difference between the number of demands and the number of stocks that can be shipped is calculated for the metaplayer (“cost +” in the figure). = abs (b) "). It also determines whether the day of the week in the current step is Saturday or Sunday (“if self.sat_sun_day_flag (y):” in the figure). Then, if it is Saturday or Sunday, the difference between the required man-hours and the man-hours that can be shipped is multiplied by the coefficient for Saturday and Sunday (Saturday-Sunday operation coefficient) to obtain the cost (“cost +” in the figure). = abs (f) * self.manpower_cost_holiday "). On the other hand, if the day is a weekday, the difference between the required manpower and the manpower that can be shipped is multiplied by the weekday coefficient (weekday operating coefficient) to obtain the cost (“cost + = abs” in the figure). (f) * self.manpower_cost_weekday "). It should be noted that these costs are an example of sales promotion costs determined according to the day of the week on which the product sales promotion is carried out. Then, the negative value of the calculated cost is used as the reward. The meta player aims to achieve the goal condition described later based on the reward set in this way.

Further, as shown in FIG. 13B, in one time step, the date is first advanced by one day (“self.step_date + = datetime.timedelta (days = 1)” in the figure). Next, it is determined whether or not the current date is the implementation date of the sales promotion (“t = self.five_six_day_flag (self.step_date.day)” in the figure). Next, specify the day of the week of the current day (“y = self.step_date.weekday ()” in the figure). Then, the number of demands for the product is determined using the sales promotion implementation date (t), the day of the week (y), and the moving average (dave) (“d = self.demand_by_date (t, y, dave)” in the figure). .. Next, the number of shipments per hour is determined (“r = self.avg_productivity” in the figure). Then, the man-hours required to ship the products for the number of demands are calculated (“n = int (d / r)” in the figure). Next, identify the man-hour selected in the action (“m = action” in the figure). Then, the number of stocks that can be shipped is calculated according to the man-hour selected in the action (“p = m * r” in the figure). In addition, the difference (f) between the man-hours (n) required to ship the products for the number of demands and the man-hours (m) that can handle the shipment is calculated (“f = n − m” in the figure). .. As can be seen from FIGS. 13A and 13B, the reward is calculated using this difference (f). Then, the cumulative difference between the number of demands and the number of stocks that can be shipped is calculated (“b = self.state [B] + (d − p)” in the figure). In this way, each state variable is changed, and in the next time step using the changed state variable, the goods are shipped based on the generated demand.

FIG. 13C shows an example of the goal condition in the simulation in this modified example. As shown in FIG. 13C, the average number of shipments with respect to the number of demands in the most recent fixed period (for example, 100 days) is calculated from the current time step (“fill rate = np.mean (shipped / demand)” in the figure). Then, when the calculated average is larger than the above-mentioned FR threshold value (“if fillrate> self.fillrate_threshold:” in the figure), it is assumed that the calculated average satisfies the Filllate condition (“fill_flag = True” in the figure). .. In addition, the sum of man-hours n and m for each day in the most recent fixed period (for example, 100 days) is calculated from the current time step (“ssum = np.sum (history, axis = 0)” in the figure). In addition, the monthly total of man-hours n required to ship products for the number of demands in the current time step is calculated (“nsum = int (ssum [N] /self.months)” in the figure). .. In addition, the total number of man-hours m selected by the action per month is calculated (“msum = int (ssum [M] /self.months)” in the figure). Then, when the calculated monthly total of man-hour m is less than or equal to the calculated monthly total of man-hour n (“if nsum> = msum:” in the figure), it is assumed that the cost minimization has been achieved. It is assumed that the time condition is satisfied (“sum_flag = True” in the figure). Then, when both of the above two conditions are satisfied, it is considered that the goal condition has been achieved (“done = fill_flag & sum_flag” in the figure).

In this modification, the control unit 11 generates a behavior model by reinforcement learning that calculates the above reward for each of a plurality of man-hours, and performs a simulation using the generated behavior model. Specifically, the control unit 11 changes the man-hours required for shipping from the lower limit to the upper limit in one day (one step) as an example, and increases the man-hours from 0 to the upper limit value that can be selected by the action by a predetermined man-hour. While repeating the simulation. Then, as an example, the control unit 11 sets the upper limit of the number of time steps of one episode to 1000 steps as described above, and when the meta player completes the action of the 1000th step time step without achieving the above goal condition. As the end of the episode, the average value of 100 episodes is calculated for the number of time steps until the goal condition is achieved for each man-hour and the FR value based on the result of executing 100 episodes for each man-hour. Then, the control unit 11 outputs the calculation result by storing the calculation result in the storage unit 12, displaying it on the display unit 14, and transmitting it from the communication unit 15 to the external device.

The user can confirm the calculation result by the prediction device 1 and determine the man-hours for the shipment of the product subject to the simulation on the sales promotion implementation date. Therefore, according to the first embodiment, the man-hours assigned to the shipment of the goods are adjusted based on the calculation result by the simulation by the prediction device 1. This balances the conflicting goals of maximizing sales and minimizing shipping costs by shipping to meet the number of demands that fluctuates due to sales promotion and keeping man-hours low in the warehouse where products are stored. It is possible to optimize the man-hours assigned to the shipment of the product on the sales promotion implementation date set in this simulation.

(Modification 2)
In the second modification, the immovable inventory in which the number of inventories does not fluctuate is specified for the products sold on the EC site, and the necessity of recommending the products is determined. In the simulation executed in this modified example, one time step is set to one day, and the user searches the information indicating the product page of the EC site on the Internet using the information processing terminal, and from the search result page. It is assumed that you will move to the product page of the EC site. In addition, the number of times the product page information is displayed in the product information search result is defined as the number of times the product is displayed, and the number of times the product is moved to the product page is defined as the number of clicks. Further, it is assumed that the size that is an index of the space occupied by one product in the warehouse is predetermined.

FIG. 14A shows an example of the simulation reward set by the OpenAI Gym in the prediction device 1, and FIG. 14B shows an example of updating each state variable executed in the OpenAI Gym. The action (behavior variable) in this simulation is a threshold value for determining the necessity of product recommendation, and for example, any value from 0.0 to 1.0 in 0.1 increments is selected. Will be done. The threshold value is an example of a value indicating the ease with which a product is recommended on an EC site.

As shown in FIG. 14A, in the simulation executed in this modified example, the number of clicks on the product page (“click_num” in the figure) and the product page in a fixed period, that is, in a predetermined number of consecutive multiple steps, are applied to the meta player. Based on the number of impressions ("view_num" in the figure), the click-through rate of the product page for the period is calculated ("click_through_rate = click_num / view_num" in the figure). Then, the value obtained by multiplying the click-through rate by the profit and the size is used as the reward (“reward = click_through_rate * profit * size” in the figure). That is, the greater the profit or size, the greater the reward. The meta player aims to achieve the goal condition described later based on the reward set in this way.

Further, as shown in FIG. 14B, in one time step, the profit per product (as an example, the amount obtained by subtracting the purchase price and the inventory cost from the selling price) (“profit” in the figure) and the product in one day. The number of page clicks (“click_num_one_day” in the figure), the number of impressions of the product page in one day (“view_num_one_day” in the figure), and the number of days in stock (“stock_days” in the figure), which is the number of days the product is stored in the warehouse. The state variables are the number of stocks (“stock_num” in the figure) and the threshold value (“boost_value” in the figure) selected by the above action. Further, in this modified example, information indicating a value other than "boost_value" in the figure is stored in the product information DB 2. Further, it is assumed that each of the above values is stored in the product information DB 2 day by day over a certain period in the past, for example, the past 30 days. The prediction device 1 acquires information of each value stored in the product information DB 2 over a certain period of time, and each time the time step advances, each value of the next day is specified based on the acquired information, and the specified value is specified. Update each state variable with. In this way, each state variable is changed, and the necessity of product recommendation in the next time step is determined using the changed state variable.

FIG. 14C shows an example of the goal condition in the simulation in this modified example. As shown in FIG. 14C, when the click-through rate in the above-mentioned fixed period is larger than the preset threshold value (“if click_through_rate> = click_through_rate _threshold:” in the figure), it is assumed that the click-through rate condition is satisfied ( In the figure, "click_through_rate_flag"
= True "). Also, if the average inventory days "stock_days" of a product for the most recent fixed period (for example, 100 days) from the current time step is larger than a preset threshold ("if stock_days> = stock_days_threshold:" in the figure), the product It is assumed that the condition of the average number of days in stock is satisfied (“stock_days_flag = True” in the figure). Then, when both of the above two conditions are satisfied, it is considered that the goal condition has been achieved (“done = click_through_rate_flag” in the figure.
& stock_days_flag ").

In this modification, the control unit 11 determines the reward based on the acquired product information, the product size, the product sales, the product purchase price, and the inventory cost determined according to the product size. Is generated by reinforcement learning that is calculated for each of a plurality of values indicating the ease of product recommendation, and a simulation is performed using the generated behavior model. Specifically, the control unit 11 repeats the simulation while changing the threshold value for determining the necessity of product recommendation from the lower limit to the upper limit and increasing the threshold value from 0.0 to 1.0 by 0.1, for example. Here, the control unit 11 sets the number of time steps of one episode as the number of days of the target period of the above-mentioned information of each value acquired from the product information DB2, and based on the result of executing 100 episodes for each threshold value, each The average value of 100 episodes is calculated for the number of time steps, the click-through rate, and the average number of days in stock until the goal condition is achieved with respect to the threshold value. For example, when the information of each of the above values over the past 30 days is acquired from the product information DB2, the number of time steps in one episode is 30. The control unit 11 outputs the calculation result by storing the calculation result in the storage unit 12, displaying it on the display unit 14, and transmitting it from the communication unit 15 to the external device.

The user can confirm the calculation result by the prediction device 1 and determine whether or not the recommendation of the product targeted for the simulation is necessary. Therefore, according to the first modification, the necessity of recommending the product on the EC site is determined based on the calculation result by the simulation by the prediction device 1. As a result, products that exceed the inventory days threshold are recommended on the EC site, that is, displayed on the user's information processing terminal. Then, if the situation in which the user does not operate on the recommended product, such as when the user does not click the displayed product, continues, the product is not recommended. As a result, the product that cannot be sold because it is not recommended and the product that cannot be sold even if it is recommended are determined, and the product that is more likely to be sold is recommended, thereby optimizing the recommendation of the product on the EC site. Can be planned.

(Modification 3)
In the third modification, the incentive given to the user is determined by changing the designated delivery date when the user orders the product for the product sold on the EC site. Here, the incentive is a point that can be used on the EC site that is returned to the user for the order of the product. Since the incentives are well known, detailed explanations will be omitted. The action (behavior variable) in this modification is an incentive, and for example, any value from 0.0 to 1.0 in increments of 0.1 is selected. Further, in the simulation executed in this modification, it is assumed that one time step is one day and the user can change the designated delivery date when ordering the product on the EC site.

FIG. 15A shows an example of the simulation reward set by the OpenAI Gym in the prediction device 1, and FIG. 15B shows an example of updating each state variable executed in the OpenAI Gym.

As shown in FIG. 15A, in the simulation executed in this modification, the man-hour value required for delivery of the product to be simulated is used as a reward (“reward = click_through_rate * profit * size” in the figure). That is, the greater the profit or size, the greater the reward. The meta player aims to achieve the goal condition based on the reward set in this way.

Further, as shown in FIG. 15B, in one time step, the order information of the product (order ID, latitude and longitude of the delivery destination, the specified delivery date and time, which is the current specified delivery date and time, and the individual delivery port. Including number) (In the figure, "order_info: (order_id, latitude, longitude,
date, time, parcel_num) ”), spatiotemporal cluster information in the delivery of goods (current specified delivery date and delivery time zone, latitude and longitude of the center position of the cluster that delivers the goods, radius of the cluster, of the cluster Number of delivery destinations that can be delivered, number of delivery destinations that are currently scheduled for delivery within the cluster) ("time_space_cluster_info: [(date, time, latitude, longitude, radius, max, num), ....]" in the figure. ), The total number of people required for delivery (“man_hour” in the figure), the total number of delivery personnel required for delivery (“deivers_num” in the figure), the total number of clusters (“cluster_num” in the figure), and the time required for delivery. When a certain total waiting time (“idle_time” in the figure), total delivery distance (“distance” in the figure), difference between the specified delivery date and time and the optimized date and time (“difference” in the figure), and the specified delivery date are changed. The incentive given to the user (as an example, the number of points. “Incentive: basic_point * action (0.0 to 1.0)” in the figure) is a state variable. Further, in this modified example, information indicating a value other than "incentive: basic_point * action (0.0 to 1.0)" in the figure is stored in the product information DB2. A cluster is an example of a delivery area of a delivery company. The prediction device 1 acquires information of each value stored in the product information DB 2 over a certain period of time, and each time the time step advances, each value of the next day is specified based on the acquired information, and the specified value is specified. Update each state variable with. In this way, each state variable is changed, and the changed state variable is used to calculate the total man-hours required for delivery in the next time step.

FIG. 15C shows an example of the goal condition in the simulation in this modified example. As shown in FIG. 15C, the total man-hours required for delivery (“man_hour_sum” in the figure) and the maximum man-hours that can be assigned to the delivery in the most recent fixed period (for example, 100 days) from the current time step (“man_hour_sum” in the figure). max_ man_hour_sum ") and calculate. Then, based on each calculated man-hour, the operating rate of the delivery staff in the relevant period is calculated (“man_hour_rate = man_hour / max_ man_hour_sum” in the figure). Then, if the calculated utilization rate of the delivery staff is less than or equal to the preset threshold value (“if man_hour_rate <= man_hour_rate_threshold:” in the figure), it is assumed that the condition of the utilization rate of the delivery staff is satisfied (“man_hour_rate_flag =” in the figure). True "). If the value of the incentive (“incentive”) for the period is less than or equal to the preset threshold value (“if incentive <= incentive_threshold:” in the figure), the incentive condition is satisfied (
In the figure, "incentive_flag = True"). Then, when both of the above two conditions are satisfied, it is considered that the goal condition has been achieved (“done = man_hour_rate_flag & incentive_flag” in the figure).

In this modification, the control unit 11 generates a behavior model by reinforcement learning that calculates a reward determined based on the total number of people required for delivery calculated from the acquired product information for each of a plurality of incentives. , Perform a simulation using the generated behavior model. Specifically, the control unit 11 repeats the simulation while changing the value indicating the ratio of the incentives from the lower limit to the upper limit and increasing the value by 0.1 from 0.0 to 1.0, for example. Here, the control unit 11 sets the number of time steps of one episode as the number of days of the target period of the above-mentioned information of each value acquired from the product information DB2, and based on the result of executing 100 episodes with respect to the ratio of each incentive. , Calculate the average value of 100 episodes for the number of time steps to achieve the goal condition for the ratio of each incentive and the occupancy rate of the delivery staff. For example, when the information of each of the above values over the past 30 days is acquired from the product information DB2, the number of time steps in one episode is 30. The control unit 11 outputs the calculation result by storing the calculation result in the storage unit 12, displaying it on the display unit 14, and transmitting it from the communication unit 15 to the external device.

When the user confirms the calculation result by the prediction device 1 and orders the product to be simulated on the EC site, the user is given an incentive to be given to the user by changing the designated delivery date of the product. The size can be determined. Therefore, according to the first modification, the magnitude of the incentive is adjusted at the EC site based on the calculation result by the simulation by the prediction device 1. In this way, while balancing the two goals of minimizing delivery costs and minimizing incentive costs, the size of the incentive given when prompting the user to change the delivery date of the product on the EC site is optimized. be able to.

1 Predictor 11 Control unit 2 Product information DB

Claims (5)

The acquisition department that acquires product information of e-commerce products,
The first behavior model and the first behavior model in which different rewards are calculated by using the acquired product information and the state variable of the environment for a plurality of values of the behavior variables allowed in the environment related to the distribution of the products. It is characterized by having a control unit that generates a third behavior model based on the strengthening learning of the second behavior model and predicts the optimum value of the behavior variable in the environment using the third behavior model. Predictor to do. The behavior variable is the number of orders for the product by the seller of the product.
The product information acquired by the acquisition unit is information indicating changes in the number of orders, the number of shipments, and the number of arrivals of the product over a certain period of time.
The environmental state variables are the sales of the product, the purchase price of the product, the shortage cost related to the shortage of the product, the sales promotion cost related to the promotion of the product, and the excess inventory of the product. The inventory cost related to and the delivery cost related to the delivery of the goods.
The control unit uses the acquired product information and the state variable of the environment to calculate different rewards for each of the number of orders corresponding to the plurality of values of the behavior variable with the first behavior model. The prediction according to claim 1, wherein the third behavior model is generated based on the reinforcement learning of the second behavior model, and the optimum number of orders is predicted using the third behavior model. apparatus. The behavioral variable is the man-hours associated with the storage of the goods in the warehouse.
The product information acquired by the acquisition unit is information indicating an implementation date of sales promotion of the product.
The state variable of the environment is a sales promotion cost determined according to the day of the week on which the sales promotion of the product is carried out.
The control unit uses the acquired product information and the state variable of the environment to calculate different rewards for each of the human times corresponding to the plurality of values of the behavior variable with the first behavior model. The prediction according to claim 1, wherein the third behavior model is generated based on the reinforcement learning of the second behavior model, and the optimum human time is predicted using the third behavior model. apparatus. The behavior variable is a value indicating the ease with which the product is recommended on the EC site.
The product information acquired by the acquisition unit is information indicating the number of clicks and the number of impressions of the product over a certain period on the EC site.
The state variables of the environment are the size of the product, the sales of the product, the purchase price of the product, and the inventory cost determined according to the size of the product.
The control unit uses the acquired product information and the state variable of the environment to give different rewards to each of the values indicating the ease of recommendation of the product corresponding to the plurality of values of the behavior variable. The third behavior model is generated based on the calculated reinforcement learning of the first behavior model and the second behavior model, and the ease of recommendation of the product is shown using the third behavior model. The prediction device according to claim 1, further comprising predicting an optimum value. The behavior variable is an incentive given to the user by changing the designated delivery date of the product on the EC site.
The product information acquired by the acquisition unit includes the delivery destination and delivery date and time of the product, the delivery area and the number of deliverable destinations of the delivery company of the product, the time required for delivery of the product, and the product. It is information indicating the delivery distance in the delivery of
The control unit uses the acquired product information and the state variable of the environment to calculate different rewards for each of the incentives corresponding to the plurality of values of the behavior variable, and the first behavior model and the above. The prediction device according to claim 1, wherein the third behavior model is generated based on the reinforcement learning of the second behavior model, and the optimum incentive is predicted using the third behavior model.

Images