CN113592240B - An MTO enterprise order processing method and system - Google Patents

Abstract

The embodiment of the invention provides an order processing method and system for an MTO enterprise, comprising the following steps: when the current order arrives at an order-oriented production (MTO) enterprise, establishing an order acceptance strategy model based on a Markov Decision Process (MDP) theory for determining an optimal strategy according to a current order queue of the MTO enterprise and the current arriving order; converting the order receiving strategy model based on the MDP theory according to the post-state theory of the reinforcement learning algorithm to obtain an MDP model based on the post-state; the difficulty in solving the MDP model based on the rear state is reduced through the learning process of the learning parameter vector of the reinforcement learning algorithm, and the MDP optimization model based on the rear state is obtained; and solving the MDP optimization model based on the rear state by adopting a three-layer artificial neural network to determine an optimal strategy for whether the current arriving order is accepted. Modeling of the dynamic change arrival of the order is more consistent with the actual state of the order in reality.

Description

An MTO enterprise order processing method and system

Technical field

The invention relates to the field of order acceptance optimization, and specifically relates to an MTO enterprise order processing method and system.

Background technique

As customer demand continues to become more personalized, more and more companies are beginning to adopt the make-to-order (MTO) model to more easily observe and contact end customers and maximize customer satisfaction. . The so-called MTO model refers to an enterprise that produces according to customer orders. Different customers have different needs for the type of orders. MTO enterprises organize and produce orders according to the order requirements put forward by customers. Under normal circumstances, the production capacity of MTO companies is limited, and coupled with various cost factors, it is impossible for companies to accept all randomly arriving customer orders, which requires MTO companies to formulate corresponding order acceptance strategies. Therefore, studying how MTO companies make order selection decisions within limited resources plays a huge role in MTO companies making full use of limited resources and maximizing long-term profits.

In the process of implementing the present invention, the applicant found that there are at least the following problems in the prior art: the strategy model is too simplified and difficult to approach the real situation.

Contents of the invention

Embodiments of the present invention provide an MTO enterprise order processing method and system. The modeling of dynamically changing order arrivals is more consistent with the status of actual orders in reality.

To achieve the above objectives, on the one hand, embodiments of the present invention provide an MTO enterprise order processing method, including:

When a current order arrives at an order-oriented production MTO enterprise, an order acceptance strategy model based on the Markov decision process MDP theory is established based on the current order queue of the MTO enterprise and the currently arriving orders. The order based on the MDP theory The acceptance strategy model is used to determine the optimal strategy; among them, the current arrival order represents the order that the MTO enterprise has received but has not yet decided whether to accept; the strategy is to accept the current arrival order or reject the current arrival order, and the optimal strategy refers to When choosing this strategy, the interests of the MTO company are optimal;

The order acceptance strategy model based on the MDP theory is transformed according to the post-state theory of the reinforcement learning algorithm, and the MDP model based on the post-state is obtained after the transformation; the solution of the MDP model based on the post-state is reduced through the learning process of the learning parameter vector of the reinforcement learning algorithm. Difficulty, obtain the MDP optimization model based on the post-state;

A three-layer artificial neural network is used to solve the MDP optimization model based on the post-state to obtain the solution result. Based on the solution result, the optimal strategy for accepting the current arriving order is determined.

On the other hand, embodiments of the present invention provide an MTO enterprise order processing system, including:

The model building unit is used to establish an order acceptance strategy model based on the Markov decision process MDP theory based on the current order queue of the MTO enterprise and the currently arriving orders when the current order arrives at the order-oriented production MTO enterprise. The above-mentioned order acceptance strategy model based on MDP theory is used to determine the optimal strategy; among them, the current arrival order represents the order that the MTO enterprise has received but has not yet decided whether to accept; the strategy is to accept the current arrival order or reject the current arrival order, so The above-mentioned optimal strategy means that the interests of the MTO enterprise are optimal when choosing this strategy;

The model transformation unit is used to transform the order acceptance strategy model based on the MDP theory according to the post-state theory of the reinforcement learning algorithm. After the transformation, an MDP model based on the post-state is obtained;

The model optimization unit is used to reduce the difficulty of solving the post-state-based MDP model through the learning process of the learning parameter vector of the reinforcement learning algorithm, and obtain the post-state-based MDP optimization model;

The solving unit is used to use a three-layer artificial neural network to solve the MDP optimization model based on the post-state to obtain the solution results, and determine the optimal strategy for accepting the current arriving order based on the solution results.

The above technical solution has the following beneficial effects: the modeling of the dynamic change of orders is more consistent with the status of actual orders in reality.

Description of the drawings

In order to explain the embodiments of the present invention or the technical solutions in the prior art more clearly, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings in the following description are only These are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without exerting creative efforts.

Figure 1 is a flow chart of an MTO enterprise order processing method according to an embodiment of the present invention;

Figure 2 is a structural diagram of an MTO enterprise order processing system according to an embodiment of the present invention;

Figure 3 is the three-layer neural network architecture;

Figure 4 is the sample learning rate;

Figure 5 shows the production capacity of different units;

Figure 6 shows the average profit of different order arrival rates;

Figure 7 shows the acceptance rate of different order arrival rates;

Figure 8 shows the average profit under different inventory costs;

Figure 9 shows the order acceptance rate under different inventory costs;

Figure 10 shows customer priority factors.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present invention.

As shown in Figure 1, combined with the embodiment of the present invention, an MTO enterprise order processing method is provided, including:

S101: When a current order arrives at an order-oriented production MTO enterprise, establish an order acceptance strategy model based on the Markov decision process MDP theory based on the current order queue and currently arriving orders of the MTO enterprise. The said order acceptance strategy model is based on the MDP theory. The order acceptance strategy model is used to determine the optimal strategy; where the current arriving order represents an order that the MTO enterprise has received but has not yet decided whether to accept; the strategy is to accept the current arriving order or reject the current arriving order, and the optimal strategy It means that the interests of the MTO enterprise are optimal when choosing this strategy;

S102: Transform the order acceptance strategy model based on MDP theory according to the post-state theory of the reinforcement learning algorithm, and obtain the MDP model based on the post-state after transformation;

S103: Reduce the difficulty of solving the post-state-based MDP model through the learning process of the learning parameter vector of the reinforcement learning algorithm, and obtain the post-state-based MDP optimization model;

S104: Use a three-layer artificial neural network to solve the MDP optimization model based on the post-state to obtain the solution result, and determine the optimal strategy for accepting the current arriving order based on the solution result.

Preferably, step 101 specifically includes:

Assume that each order is not split into production, that the order is shipped to the customer in one go after production is completed, and that once the MTO company accepts the order, the order cannot be changed or canceled;

Determine the information of each order in the current order queue and the current arriving order based on the current order queue and the current arriving order. The order information includes: the customer priority μ corresponding to the order, the unit product price pr, the required product quantity q, the lead time lt and the minimum Late delivery date dt; among them, the order in the current order queue is an accepted order, the order arrival obeys the Poisson distribution with parameter λ, and the unit product price and product demand quantity in the order obey uniform distribution respectively;

Determine the reward sub-items faced based on the information of each order in the current order queue and the current arriving order. The reported sub-items include: rejection cost of rejecting the current arriving order, profit of accepting the currently arriving order, and delay penalty for orders in the current order queue. Cost, the inventory cost of the order in the current order queue; where:

If the currently arriving order is rejected, a rejection cost will be incurred: μ*J, where J represents the rejection cost without considering customer priority;

If the current arrival order is accepted, the profit of the order I will be obtained: I=pr*q, and the production cost C will be consumed at the same time: C=c*q, where c is the unit product production cost;

For each order in the current order queue, the MTO company will produce according to the principle of first come, first served. If there is a delayed order in the current order queue, and the delivery time of the delayed order is within the latest delivery period, the MTO company will The delay penalty cost Y is paid to the customer corresponding to the delayed order: Among them, t represents the production time still required for the accepted order, b represents the unit production capacity of the MTO enterprise, and u represents the unit product delay penalty cost per unit time of the MTO enterprise;

If the product of the order in the current order queue is generated within the lead time and the product is temporarily stored in the warehouse, the product is temporarily stored in the MTO enterprise warehouse, resulting in inventory cost N: Among them, h represents the inventory cost per unit product per unit time;

According to the Markov decision process MDP theory, as well as the information and return items of each order in the current order queue and the current arriving order, an order acceptance strategy model based on the MDP theory of the MTO enterprise is established. The MTO enterprise's order acceptance strategy model is based on the MDP theory. The order acceptance strategy model is a four-tuple (S, A, f, R), which are the state space S, the action space A, the state transition function f, and the reward function R, where:

The state space S represents the state of the system where the MTO enterprise order processing method is located; the state space S is an nx6-dimensional vector, where n represents the number of order types, and 6 represents 6 types of order information: customer priority μ, unit product price pr, Product demand quantity q, lead time lt, latest delivery date dt still requires production completion time t for orders in the current order queue, where t has a preset maximum upper limit value;

Action space A represents the action set for the currently arriving order; when the current order arrives at time m, the MTO enterprise needs to make an action decision of accepting the order or rejecting the order, and the action set of accepting the order or rejecting the order forms the action space amount A, A=(a ₁ , a ₂ ), where a ₁ represents accepting the order and a ₂ represents rejecting the order;

The state transition function f represents the transition from the current state to the m decision time state, and the m decision time refers to the time when action is taken for the current arriving order; the generation process of the state transition function f is:

Assume that the order information μ, pr, q, lt, and dt are all independent and identically distributed. According to the initial state s and the action a taken at decision time m, the probability density function f of the state at the next decision time m+1 is (·|(s, a)) are respectively expressed as f _M (x), f _PR (x), f _Q (x), f _LT (x), f _DT (x);

And the order information μ _m+1 , pr _m+1 , q _m+1 , lt _m+1 , dt _m+1 at the next decision time m+1 are independent with respect to (s _m , a _m ); where, t _m+1 is expressed as:

Equation (1) indicates that t _m+1 is affected by (s _m , a _m ), and different (q _m , t _m , a _m ) lead to different order production times, and t _m+1 is also affected by the order arrival time. The influence of interval; among them, AT _m→m+1 represents the time interval between the arrival of two orders, that is, according to the arrival of each order, it obeys the Poisson distribution with parameter λ;

According to the current state s, action a, and order information μ _m+1 , pr _m+1 , q _m+1 , lt _m+1 , dt _m+1 are independent characteristics about (s _m , a _m ), we get the following The conditional probability density of state s′ at a decision moment m+1, and the conditional probability density of state s′ at m+1 at the next decision moment are expressed as:

f(s′|s,a)＝f _M (μ′)*f _PR (pr′)*f _Q (q′)*f _LT (lt′)*f _DT (dt′)*f _T (t′ |s,a)

where f _T (t′|s, a) represents the production time still required to produce the accepted order at the next decision moment m+1 after taking action a in the current state s, f _T (t′|s, a ) is defined by formula (1) and related random variables;

When the MTO enterprise makes a decision at m, the reward function R represents the corresponding reward obtained after taking actions on the currently arrived order. The reward function R is expressed as:

Among them, when a _m =1 means that if the MTO enterprise receives the current arriving order, the reward function R is expressed as ICYN;

When a _m = 0, it means that if the MTO enterprise rejects the current arriving order, the reward function R is expressed as -μ*J;

For any strategy in the order acceptance strategy model based on MDP theory, its corresponding value function is defined according to the reward function, and the average long-term profit corresponding to the strategy is represented by the value function. The value function is expressed as:

Among them, π represents any strategy, γ represents future reward discount, 0 < γ ≤ 1, and by setting γ to ensure that the summation term defined by the formula is meaningful, n represents the total number of decision-making moments, and each current order corresponds to a decision-making moment;

The optimal strategy π ^* is determined by the average long-term profit of any strategy π. The optimal strategy π ^* is used to ensure the long-term profit maximization of the enterprise. At this time, the interests of the MTO enterprise are optimal; the optimal strategy π ^* is expressed as:

Among them, Π represents the set of all strategies.

Preferably, step 102 specifically includes:

After receiving the current arriving order at time m and making a decision, the post-state variable p _m is set according to the reinforcement learning algorithm. The post-state variable represents the production time still needed to produce the accepted order after selecting action a _m at decision time m; where, The post-state is an intermediate variable between two consecutive states;

The post-state variable p _m is determined according to the current state s _m and action a _m , and the post-state variable p _m is expressed as:

According to p _m, the production time t m+1 still required for the order that has been accepted at the next decision time _m+1 is expressed as:

Among them, in formula (5), Indicates the larger number between the variables , dt′, t′) conditional probability density is expressed as:

Among them, the conditional probability density function f _T (·|p) is defined by equation (5) and related random variables;

After setting the post-state variable, rewrite the conditional expectation E[] in the MDP theory, and after rewriting the conditional expectation E[], define the value function of the post-state, and express the value function of the post-state as:

J ^* (p)＝γE[V ^* (s′)|p] (7)

The optimal strategy π ^* is constructed through the post-state value function, thereby turning the optimal strategy π ^* into a one-dimensional state space. The optimal strategy π ^* is expressed as:

Preferably, step 103 specifically includes:

The reinforcement algorithm does not directly calculate J ^* when constructing the optimal policy ^{π * through} the post-state value function J * for solving, but adopts the learning process of learning parameter vectors to realize the approximation of J ^* for solving; adopts the method of learning parameter vectors The learning process is used to achieve the approximate solution of J ^* , as follows:

Determined according to the given parameter vector θ as well as

The parameter vector θ is used to perform parameter learning from data samples using the reinforcement algorithm to obtain the parameter vector θ ^* , and the parameter vector θ ^* obtained through learning is determined. To approximate J ^* , determine the optimal strategy π ^* based on the approximate J ^* .

Preferably, step 104 specifically includes:

J ^* (p) is solved through a three-layer artificial neural network ANN, and J ^* (p) is approximated to arbitrary precision. Using a three-layer artificial neural network ANN is expressed as:

Among them, the parameter vector can be expressed as:

θ=[w ₁ ,...,w _N ,α ₁ ,...α _N ,u ₁ ,...u _N ,β]

Φ _H (x)＝1/(1+e ^-x )

Function of equation (11) It is a three-layer single-input single-output neural network. It has only a single-node input layer, whose output represents the value of the post-state p, and a hidden layer containing N nodes. The input of the i-th node is the weighted post-state. The sum of the value w _i *p and the hidden layer bias α _i ; and the input-output relationship of each hidden layer node is represented by the Φ _H (·) function, where Φ _H (·) is called the activation function; the output layer There is a node. The input of the output layer is the sum of the weighted output of the hidden layer and the output layer bias β. The output of the output layer represents the final approximate function value/>

As shown in Figure 2, combined with the embodiment of the present invention, an MTO enterprise order processing system is provided, including:

The model building unit 21 is used to establish an order acceptance strategy model based on the Markov decision process MDP theory based on the current order queue of the MTO enterprise and the currently arriving orders after the current order arrives at the order-oriented production MTO enterprise. The order acceptance strategy model based on MDP theory is used to determine the optimal strategy; where the current arrival order represents an order that the MTO enterprise has received but has not yet decided whether to accept; the strategy is to accept the current arrival order or reject the current arrival order, The optimal strategy refers to the best interests of the MTO enterprise when choosing this strategy;

The model transformation unit 22 is used to transform the order acceptance strategy model based on the MDP theory according to the post-state theory of the reinforcement learning algorithm, and obtain an MDP model based on the post-state after transformation;

The model optimization unit 23 is used to reduce the difficulty of solving the post-state-based MDP model through the learning process of the learning parameter vector of the reinforcement learning algorithm, and obtain the post-state-based MDP optimization model;

The solving unit 24 is used to use a three-layer artificial neural network to solve the post-state-based MDP optimization model to obtain a solution result, and determine the optimal strategy for whether to accept the current arriving order based on the solution result.

Preferably, the model building unit 21 is specifically used for:

Assume that each order is not split into production, that the order is shipped to the customer in one go after production is completed, and that once the MTO company accepts the order, the order cannot be changed or canceled;

Determine the information of each order in the current order queue and the current arriving order based on the current order queue and the current arriving order. The order information includes: the customer priority μ corresponding to the order, the unit product price pr, the required product quantity q, the lead time lt and the minimum Late delivery date dt; among them, the order in the current order queue is an accepted order, the order arrival obeys the Poisson distribution with parameter λ, and the unit product price and product demand quantity in the order obey uniform distribution respectively;

Determine the reward sub-items faced based on the information of each order in the current order queue and the current arriving order. The reported sub-items include: rejection cost of rejecting the current arriving order, profit of accepting the currently arriving order, and delay penalty for orders in the current order queue. Cost, the inventory cost of the order in the current order queue; where:

If the currently arriving order is rejected, a rejection cost will be incurred: μ*J, where J represents the rejection cost without considering customer priority;

If the current arrival order is accepted, the profit of the order I will be obtained: I=pr*q, and the production cost C will be consumed at the same time: C=c*q, where c is the unit product production cost;

For each order in the current order queue, the MTO company will produce according to the principle of first come, first served. If there is a delayed order in the current order queue, and the delivery time of the delayed order is within the latest delivery period, the MTO company will The delay penalty cost Y is paid to the customer corresponding to the delayed order: Among them, t represents the production time still required for the accepted order, b represents the unit production capacity of the MTO enterprise, and u represents the unit product delay penalty cost per unit time of the MTO enterprise;

If the product of the order in the current order queue is generated within the lead time and the product is temporarily stored in the warehouse, the product is temporarily stored in the MTO enterprise warehouse, resulting in inventory cost N: Among them, h represents the inventory cost per unit product per unit time;

According to the Markov decision process MDP theory, as well as the information and return items of each order in the current order queue and the current arriving order, an order acceptance strategy model based on the MDP theory of the MTO enterprise is established. The MTO enterprise's order acceptance strategy model is based on the MDP theory. The order acceptance strategy model is a four-tuple (S, A, f, R), which are the state space S, the action space A, the state transition function f, and the reward function R, where:

The state space S represents the state of the system where the MTO enterprise order processing method is located; the state space S is an nx6-dimensional vector, where n represents the number of order types, and 6 represents 6 types of order information: customer priority μ, unit product price pr, Product demand quantity q, lead time lt, latest delivery date dt still requires production completion time t for orders in the current order queue, where t has a preset maximum upper limit value;

Action space A represents the action set for the currently arriving order; when the current order arrives at time m, the MTO enterprise needs to make an action decision of accepting the order or rejecting the order, and the action set of accepting the order or rejecting the order forms the action space amount A, A=(a ₁ , a ₂ ), where a ₁ represents accepting the order and a ₂ represents rejecting the order;

The state transition function f represents the transition from the current state to the m decision time state, and the m decision time refers to the time when action is taken for the current arriving order; the generation process of the state transition function f is:

Assume that the order information μ, pr, q, lt, and dt are all independent and identically distributed. According to the initial state s and the action a taken at decision time m, the probability density function f of the state at the next decision time m+1 is (·|(s, a)) are respectively expressed as f _M (x), f _PR (x), f _Q (x), f _LT (x), f _DT (x);

And the order information μ _m+1 , pr _m+1 , q _m+1 , lt _m+1 , dt _m+1 at the next decision time m+1 are independent with respect to (s _m , a _m ); where, t _m+1 is expressed as:

Equation (1) indicates that t _m+1 is affected by (s _m , a _m ), and different (q _m , t _m , a _m ) lead to different order production times, and t _m+1 is also affected by the order arrival time. The influence of interval; among them, AT _m→m+1 represents the time interval between the arrival of two orders, that is, according to the arrival of each order, it obeys the Poisson distribution with parameter λ;

According to the current state s, action a, and order information μ _m+1 , pr _m+1 , q _m+1 , lt _m+1 , dt _m+1 are independent characteristics about (s _m , a _m ), we get the following The conditional probability density of state s′ at a decision moment m+1, and the conditional probability density of state s′ at m+1 at the next decision moment are expressed as:

f(s′|s,a)＝f _M (μ′)*f _PR (pr′)*f _Q (q′)*f _LT (lt′)*f _DT (dt′)*f _T (t′ |s,a)

where f _T (t′|s, a) represents the production time still required to produce the accepted order at the next decision moment m+1 after taking action a in the current state s, f _T (t′|s, a ) is defined by formula (1) and related random variables;

When the MTO enterprise makes a decision at m, the reward function R represents the corresponding reward obtained after taking actions on the currently arrived order. The reward function R is expressed as:

Among them, when a _m =1 means that if the MTO enterprise receives the current arriving order, the reward function R is expressed as ICYN;

When a _m = 0, it means that if the MTO enterprise rejects the current arriving order, the reward function R is expressed as -μ*J;

For any strategy in the order acceptance strategy model based on MDP theory, its corresponding value function is defined according to the reward function, and the average long-term profit corresponding to the strategy is represented by the value function. The value function is expressed as:

Among them, π represents any strategy, γ represents future reward discount, 0 < γ ≤ 1, and by setting γ to ensure that the summation term defined by the formula is meaningful, n represents the total number of decision-making moments, and each current order corresponds to a decision-making moment;

The optimal strategy π ^* is determined by the average long-term profit of any strategy π. The optimal strategy π ^* is used to ensure the long-term profit maximization of the enterprise. At this time, the interests of the MTO enterprise are optimal; the optimal strategy π ^* is expressed as:

Among them, Π represents the set of all strategies.

Preferably, the model conversion unit 22 is specifically used for:

After receiving the current arriving order at time m and making a decision, the post-state variable p _m is set according to the reinforcement learning algorithm. The post-state variable represents the production time still needed to produce the accepted order after selecting action a m at decision time _m ; where, The post-state is an intermediate variable between two consecutive states;

The post-state variable p _m is determined according to the current state s _m and action a _m , and the post-state variable p _m is expressed as:

According to p _m, the production time t m+1 still required for the order that has been accepted at the next decision time _m+1 is expressed as:

Among them, in formula (5), Indicates the larger number between the variables , dt′, t′) conditional probability density is expressed as:

Among them, the conditional probability density function f _T (·|p) is defined by equation (5) and related random variables;

After setting the post-state variable, rewrite the conditional expectation E[] in the MDP theory, and after rewriting the conditional expectation E[], define the value function of the post-state, and express the value function of the post-state as:

J ^* (p)＝γE[V ^* (s′)|p] (7)

The optimal strategy π ^* is constructed through the post-state value function, thereby turning the optimal strategy π ^* into a one-dimensional state space. The optimal strategy π ^* is expressed as:

Preferably, the model optimization unit 23 is specifically used for:

The reinforcement algorithm does not directly calculate J ^* when constructing the optimal policy ^{π * through} the post-state value function J * for solving, but adopts the learning process of learning parameter vectors to realize the approximation of J ^* for solving; adopts the method of learning parameter vectors The learning process is used to achieve the approximate solution of J ^* , as follows:

Determined according to the given parameter vector θ as well as

The parameter vector θ is used to perform parameter learning from data samples using the reinforcement algorithm to obtain the parameter vector θ ^* , and the parameter vector θ ^* obtained through learning is determined. To approximate J ^* , determine the optimal strategy π ^* based on the approximate J ^* .

Preferably, the solving unit 24 is specifically used for:

J ^* (p) is solved through a three-layer artificial neural network ANN, and J ^* (p) is approximated to arbitrary precision. Using a three-layer artificial neural network ANN is expressed as:

Among them, the parameter vector can be expressed as:

θ=[w ₁ ,...,w _N ,α ₁ ,...α _N ,u ₁ ,...u _N ,β]

Φ _H (x)＝1/(1+e ^-x )

Function of equation (11) It is a three-layer single-input single-output neural network. It has only a single-node input layer, whose output represents the value of the post-state p, and a hidden layer containing N nodes. The input of the i-th node is the weighted post-state. The sum of the value w _i *p and the hidden layer bias α _i ; and the input-output relationship of each hidden layer node is represented by the Φ _H (·) function, where Φ _H (·) is called the activation function; the output layer There is a node. The input of the output layer is the sum of the weighted output of the hidden layer and the output layer bias β. The output of the output layer represents the final approximate function value/>

The above technical solutions of the embodiments of the present invention will be described in detail below with reference to specific application examples. For technical details that are not introduced during the implementation process, reference can be made to the relevant descriptions above.

The present invention is a method and system for MTO enterprise order acceptance strategy based on post-state reinforcement learning. In order to solve the problems of incomplete model considerations and high complexity of the solution process in existing research, the present invention will consider the MDP model and further introduce inventory Cost and multiple customer priority order acceptance decision factors. At the same time, a low-complexity solution algorithm based on a combination of after-state and neural networks is proposed to solve the MTO enterprise order acceptance problem.

As the diversified needs of customers continue to increase, the make-to-order (MTO) model, which organizes production according to customers' different needs for orders, is becoming more and more important in corporate production activities. Due to the limitation of enterprise production capacity, MTO enterprises need to formulate reasonable order acceptance strategies, that is, how to determine whether to accept arriving orders based on production capacity and order status, in order to improve the enterprise's production efficiency.

On the basis of the traditional order acceptance problem, the present invention proposes a more complete model of the MTO enterprise order acceptance problem: on the basis of the traditional model elements of delayed delivery cost, rejection cost, and production cost, the present invention further considers the order inventory cost. , multiple customer priority factors, and model the optimal order acceptance problem as a Markov decision process (MDP). In addition, since the classic MDP solution method relies on the solution and estimation of high-dimensional state value functions, its computational complexity is high. Therefore, in order to reduce complexity, the present invention proposes to use a one-dimensional post-state value function to replace the high-dimensional state value function, and combine it with a neural network to approximate the post-state value function. Finally, the applicability and superiority of the order acceptance strategy model and algorithm proposed by the present invention are verified through simulation.

1. Description of the problem to be solved by this invention and model assumptions

This invention assumes that an MTO enterprise with limited production capacity produces through a single production line. Assume that there are n types of customer orders in the market. Order-related information includes customer priority μ, unit product price pr (pr is the abbreviation of price), quantity q, lead time lt (full name "leadtime") and latest delivery Period dt (full name "deliverytime"), etc. The lead time lt refers to the agreed delivery time, the working time cycle of the order production under no unexpected circumstances, that is, the time from the start of the order production work to the end of the order production work; but when the enterprise cannot fully guarantee whether it can be delivered within the agreed time When completed, the preset time period will be postponed to the agreed delivery time, and the preset postponement time period will be the latest delivery date dt. Customer order arrival follows a Poisson distribution with parameter λ. The price of a single product in an order and the quantity demanded of the corresponding product are uniformly distributed.

When an order arrives, the company needs to judge whether to accept the order based on its own production capacity. If the order is rejected, a rejection cost μ*J is generated. The higher the customer priority, the higher the rejection cost; where μ represents the customer priority coefficient, and J represents the rejection cost when the customer priority is not considered (J is also order-related information) .

If the order is accepted, the profit of the order is obtained, that is, I=pr*q, and the production cost is consumed, that is, C=c*q, where c is the production cost of the unit product. MTO enterprises will produce accepted orders on a first-come, first-served basis. If the order is not delivered within the lead time required by the customer, that is, When , the enterprise needs to pay a certain delay penalty cost Y, that is/> Among them, t represents the production time still needed for the accepted order before accepting the current order, b represents the unit production capacity of the enterprise, u represents the unit product delay penalty cost per unit time, and the higher the customer priority, the higher the delay penalty cost. Customers do not take delivery of products produced before the lead time, that is, if When, the products are temporarily stored in the MTO enterprise warehouse, resulting in inventory cost N, that is,/> Where h represents the unit product inventory cost per unit time. Each order is not divided into separate productions. After the order is completed, it will be sent to the customer in one go. Once the MTO company accepts the order, the customer cannot change or cancel the order.

The problem to be solved by this invention is that when a customer order arrives randomly, the MTO enterprise makes a decision based on considering the current order queue, as well as backorder costs, rejection costs, production costs, inventory costs and various customer priority factors. Whether to accept currently arriving orders to ensure the company's long-term average profit maximization.

2. Order acceptance strategy modeling based on MDP theory

It can be seen that the MTO enterprise order acceptance decision-making problem is a type of stochastic sequential decision-making problem (or called a stochastic system multi-stage decision-making problem). After the decision-maker of the MTO enterprise decides to accept or reject the order, the system state changes, but the development process of this decision-making stage after the current stage is not affected by the states of the previous stages, that is, it has no aftereffects. Therefore, this problem can be abstracted into an MDP model based on the MDP (Markov Decision Process) theory. The MDP model is defined as a four-tuple (S, A, f, R), which respectively represents the state space S, action space A, state transition function f, and reward function R:

1) System status: Assuming that there are n order types (there are n order types) in the order acceptance system (order processing system), the system status can be represented by the vector S: S = (μ, pr, q, lt, dt, t ), t represents the production completion time required for accepted orders. Based on MTO companies with limited production capacity, t has a maximum upper limit.

2) System action set: At time m, when a customer order arrives, the MTO enterprise needs to make a decision to accept or reject the order. The action set in the model can be represented by the vector A = (a ₁ , a ₂ ), where a ₁ means accepting the order, a ₂ means rejecting the order. Among them, the actions in vector A are only for the currently arriving orders, and do not include orders in the order queue.

3) State transition model: Given the initial state s (the initial state represents the first arriving order in the order acceptance system) and the action a that has been taken, the probability density function of the next state is f(·|(s , a)) means. It is assumed here that the order information μ, pr, q, lt, and dt are all independent and identically distributed. Then the probability density function can be used to represent their respective distributions, and the corresponding probabilities of the distributions of the order information μ, pr, q, lt, and dt can be determined. The density functions are f _M (x), f _PR (x), f _Q (x), f _LT (x), f _DT (x), so the order information μ _m+1 , pr _m+1 , q _m+1 , lt _m+1 , dt _m+1 are independent with respect to (s _m , a _m ), which means that in the current state s _m , after taking the action a _m , the next decision moment (the next moment state s _{m+ 1} ) order information. But t _m+1 is affected by (s _m , a _m ). This is because different (q _m , t _m , a _m ) will lead to different order production times. t _m+1 is also affected by the order arrival time interval. The influence of t _m+1 can be expressed as:

Among them, AT _m→m+1 represents the time interval between the arrival of two orders, that is, the arrival of each order obeys the Poisson distribution with parameter λ.

Since the order information μ _m+1 , pr _m+1 , q _m+1 , lt _m+1 , dt _{m +1} are independent with respect to (s _m , am ), when the current state s and action a are given, , according to the conditional probability density of the current state, the conditional probability density of the state s′ at the next decision moment can be obtained. The conditional probability density of the state s′ at the next decision moment can be expressed as:

f(s′|s,a)＝f _M (μ′)*f _PR (pr′)*f _Q (q′)*f _LT (lt′)*f _DT (dt′)*f _T (t′ |s,a)

where f _T (t′|s, a) represents the production time still required to produce the accepted order at the next moment after taking action a in the current state s. Its specific form can be defined by (1) and related random variables. .

4) Reward function: At decision moment m, the immediate reward function obtained by the MTO enterprise after making the decision of whether to accept the order is:

Among them, I represents the profit of the order (unit product price × product quantity); C represents the production cost; Y represents the delay penalty cost (if the order is completed beyond the lead time, a delay penalty cost will be incurred); Completed before the deadline, the customer does not pick up the goods in advance, resulting in inventory costs).

After the decision-maker takes the corresponding action (reject or accept the order) in the current state, the reward function is the evaluation of the action given by the system.

5) Optimal strategy: In the MTO enterprise order acceptance problem, the goal is to find an optimal order acceptance strategy π ^* to maximize the long-term profit of the enterprise. Each policy π is a function from system state to action, which determines how the enterprise chooses whether to accept an order based on current state information. For any strategy π, define its value function as its average long-term profit (the value function represents the value of the decision-maker in state s, after choosing action a, and then following this strategy (i.e., the cumulative discounted reward), which is taken from the current state The cumulative discounted reward obtained after the action). The value function is as follows:

Among them, 0<γ≤1 represents future reward discount (which ensures that the summation term defined in (2) is meaningful). What we care about is the optimal strategy π ^* among all strategies, which is defined as:

where Π represents the set of all strategies.

The standard MDP theory is introduced as follows:

In MDP theory, the optimal policy π ^* can be constructed using the state value function y*:

The expectation E[·] is defined as the next random state s′ given the current state s and action a. At the same time, V ^* is a solution to the Bellman equation, that is:

And V ^* can be solved by value iteration.

It can be seen that the classic MDP theory provides an optimal strategy solution method based on the state value function V ^* . However, the solution of the state value function V ^* is very difficult. This is because the system state of the problem to be solved by the present invention is high-dimensional and continuous, which leads to the representation of the state value function V ^* and the value based on the expectation E[·] The computational complexity of iteration is prohibitive. In order to solve the above problems, the present invention proposes an optimal strategy construction method based on the post-state value function.

3. MDP model transformation based on post-state

The after-state is an intermediate variable between two consecutive states and can be used to simplify the optimal control of some MDPs. The concept of post-state is a technique in reinforcement learning often used for learning tasks in board games. For example, when an agent uses a reinforcement learning algorithm to play chess, the agent can control its own moves with certainty, while the opponent's moves are random. Before deciding on an action, the Agent faces a specific chess piece position on the chessboard, which is the same state as in the classic MDP model. The post-state of each move of the Agent is defined as the state of the board after this move but before the opponent moves. If the agent can understand the winning chances of all different post-states, then it can use these known probabilities to achieve optimal behavior, that is, simply choose the post-state with the highest chance of winning and take the corresponding action.

The present invention uses a similar post-state method to transform the order acceptance problem. In particular, in the order acceptance problem we consider, the post-state variable p _m is defined as the production time still required to produce the accepted order after selecting action a m at decision moment _m . Therefore, given the current state s _m and action a _m , the post-state can be expressed as:

It is not difficult to see that given p _m , the production time t _m+1 still required for the order m+1 at the next decision moment can be expressed as:

in It means taking the larger number between variable x and 0; AT is the arrival time interval between two orders. Therefore, given the current subsequent state p, the conditional probability density of the next decision moment state S′ = (μ′, p′, q′, lt′, dt′, t′) can be expressed as:

Among them, the conditional probability density function f _T (·|p) is defined by (5) and related random variables.

Therefore, the conditional expectation E[V(s′)|s, a] in equations (3) and (4) can be rewritten as the conditional expectation E[V(s′)|σ(s, a)], so π ^* can be redefined as follows. First define the post-state value function as:

J ^* (p)＝γE[V ^* (s′)|p] (7)

Adding (7) to (3), the optimal policy π ^* can be constructed using the post-state value function J ^* as follows:

Further, substituting (7) into (4), we get:

So, get:

In fact, it can be seen from (7) that γE[V ^* (s′)|p] is J ^* (p), so we can get

Finally, we solve J ^* through the reinforcement learning median iteration algorithm ^[19] , that is, J ₀ is an arbitrary initialization function, with

When k→∞, J _k converges to J ^* .

It can be seen from equation (3) that the expectation E[V ^* (s′)|s, a] needs to be used when calculating the optimal strategy, while the optimal strategy in equation (8) not only does not need to consider the expectation, but directly uses J ^* To replace E[V ^* (s′)|s, a], and reduce the high-dimensional state space to a one-dimensional state space, which greatly reduces the solution complexity.

4. Optimal control based on neural network

It has been proved before that π ^* can be constructed using J ^* , and J ^* can be solved by value iteration of formula (9). However, there are two difficulties in the implementation process of formula (9): first, if f _M (·), f _PR (·), f _Q (·), f _LT (·), f _DT (·) and f _T ( ·|σ(s, a)) is unobtainable, then E[·|p] cannot be calculated; secondly, since the post-state changes continuously, each iteration of equation (9) must be performed on an infinite number of p values Calculation. Reinforcement learning provides an effective way to solve the above two difficulties. Reinforcement learning does not directly calculate J ^* , but adopts an approximation of J ^* by learning parameter vectors, and the learning process uses data samples. In other words, the design of the RL algorithm (reinforcement learning algorithm or reinforcement learning) includes:

1) Parameterization: This determines how the function is determined from a given parameter vector θ Among them, θ represents the approximate parameter vector of the post-state value function.

2) Parameter learning: The parameter vector θ ^* is learned from a batch of data samples, and makes the slice To approximate J ^* , the optimal strategy can be expressed as:

Comparing (10) and (8), we can see that if Approximate to J ^* (p), then/>

Close to the optimal policy π ^* .

4.1 Neural network approximation

The universal approximation theorem (universal approximation theorem) shows that a three-layer artificial neural network (ANN) can approximate a continuous function to arbitrary precision. Therefore, for the J ^* (p) to be solved in this invention, ANN is a good s Choice. so It can be expressed as a neural network:

The parameter vector can be expressed as:

θ=[w ₁ ,...,w _N ,α ₁ ,...α _N ,u ₁ ,...u _N ,β],

Φ _H (x)=1/(1+e ^-x ).

The function in formula (11) As shown in Figure 3, it is actually a three-layer single-input single-output neural network. Specifically, there is only a single-node input layer, whose output represents the value of the post-state p, and a hidden layer containing N nodes. The input of the i-th node is the weighted post-state value w _i *p and the implicit The sum of layer biases α _i . The input-output relationship of each hidden layer node is represented by the Φ _H (·) function, where Φ _H (·) is called the activation function. Finally, the output layer has a node whose output represents the final approximated function value/> Its input is the sum of the hidden layer weighted output and the output layer bias β.

4.2 Value iterative training ANN (three-layer artificial neural network)

In order to achieve optimal control (that is, obtain the optimal strategy), the parameters in the three-layer artificial neural network are trained through value iteration. The specific training process is as follows.

1) Acquisition of training data: This invention requires a batch of training samples Among them, for each m, sampling obtains p _m , μ _m , pr _m , q _m , lt _m , dt _m , where p _m obeys the uniform distribution, μ _m ～f _M (·), pr _m ～f _PR (·) , q _m ~ f _Q (·), lt _m ~ f _LT (·), dt _m ~ f _DT (·). Further, t _{m is generated according to p m ,} that is, t _m =( _pm -AT) ⁺ , where AT is a random variable obeying Poisson distribution.

2) Iterative fitting (see Algorithm 1 in Table 1): As in (9), at the k-th iteration, assume the current ANN parameter vector is θ _k , and use the function defined in (11) That is, given the current value function/> The updated value function is:

We want to use the updated parameters to get a new function to approximate J _k+1 (p). So, from Γ and θ _k , we construct a batch of training data:

Among them, o _m means that at a given The expected value of J _k+1 (pm) in the case of is shown in equation (12).

Based on the training data Υ _k , parameter update can be expressed as:

Among them, L(θ|Υ _k ) is the training error:

3) Training parameters: Apply gradient descent to solve ANN parameters θ _k+1 such that The error is smallest on Υ _k , see equation (14). Gradient descent searches the parameter space iteratively: the initial parameter θ ⁽⁰⁾ in the gradient iteration is set to θ _k , then the parameters are updated during the iteration as:

Among them, α is the update step parameter, is the gradient of L at θ ^(z) defined in equation (15). Therefore, given a sufficient number of iterations Z, we use θ _k+1 = θ ^(Z) as an approximate solution to (14). Finally/> Expressed as:

Table 1 Algorithm 1 approximates J ^* (p)

In summary, the beneficial effects achieved by the present invention are as follows:

1) This invention starts from the idea of revenue management and considers the order acceptance problem of MTO enterprises in a random dynamic environment. First, on the basis of considering the enterprise's production cost, delay penalty cost and rejection cost factors, it also considers the order completion before the lead time. Inventory costs and various customer priority factors construct an MDP (Markov Decision Process) order acceptance model.

2) The present invention uses the post-state method to convert the solution of the optimal strategy in the traditional MDP, proving that the optimal strategy based on the state value function in the classic MDP problem can be equivalently defined by the value function based on the post-state. and construction, which converts multi-dimensional control problems into one-dimensional control problems, thereby greatly simplifying the solution process.

3) Traditional algorithms such as SRASA and SMART are tabular reinforcement learning methods, which can only handle optimal decision-making problems in discrete state spaces. In this regard, in order to solve the problem of order acceptance strategy learning in continuous state space, the present invention uses a neural network to parameterize the post-state value function and designs a corresponding training algorithm to realize the estimation of the post-state value function and the order acceptance strategy. Quick solution.

5. Numerical simulation experiments

The data generation method for the relevant order information required in the simulation is generated according to the following rules: the order price pr obeys the uniform distribution U(e ₁ , l ₁ ), the order quantity q obeys the uniform distribution U(e ₂ , l ₂ ), and the arrival of the order It obeys the Poisson distribution with parameter λ; there is a linear decreasing function relationship between the order lead time lt and the order price, that is, lt=δ-β*pr; the latest acceptable delivery time is an integer, and its setting satisfies the relationship expression where φ is the lead time elasticity coefficient.

Here we select pr~U[30,50], q~U[300,500], λ=0.3, δ=36, β=0.4, φ=0.8. The enterprise's unit production capacity, unit production cost, and rejection cost are respectively obtained as b=20, c=15, and J=200. The delay penalty cost per unit quantity per unit time is u=4, the customer level obeys the uniform distribution μ~U(0,1], and the inventory cost per unit quantity per unit time is h=4. Finally, the initial learning rate of the algorithm α=0.001, and the exploration rate ε= 0.1.

The simulation experiment uses Python for programming. Through the simulation experiment, the effectiveness of the MTO enterprise order acceptance strategy of the reinforcement learning value iteration neural network (value iteration with neural network) algorithm based on one-dimensional post-state space proposed by the present invention, namely: AFVINN algorithm, is verified. sex analysis.

This simulation experiment consists of two parts: In the first part, first, the learning efficiency of the AFVINN algorithm and the traditional Q-learning algorithm are compared. The comparison strategy is evaluated through the sample utilization efficiency; secondly, based on the comparison in the literature [13, 16] The strategy is to use the proposed algorithm and the FCFS method to compare and analyze the long-term average profit and order acceptance rate. The FCFS method means that when the order arrives, if the company has the ability to complete production within the latest delivery period, it will directly accept it. The order; order acceptance rate is the number of accepted orders divided by the total number of arriving orders. In the second part, we first examine the impact of the AFVINN algorithm on the average profit and order acceptance rate of MTO companies under two scenarios: considering inventory costs and not considering inventory costs; secondly, by adjusting the delay penalty cost related to customer priority and rejection cost factors to analyze the impact of the AFVINN algorithm on the average profit and order acceptance rate of MTO companies; finally, we examine the three scenarios of considering multiple customer priorities, considering three customer priorities, and not considering customer priorities. The impact of AFVINN algorithm on the average profit of MTO enterprises.

5.1 Algorithm comparison

In the existing research on reinforcement learning MTO enterprise order acceptance strategies, traditional multi-dimensional state space is used for modeling and solving. In this regard, the present invention will propose a learning efficiency comparison between the AFVINN algorithm and the traditional Q-learning algorithm. The comparison strategy is to consume 200 data samples per iteration, and evaluate the learning efficiency based on the number of data samples consumed.

It can be seen from Figure 4: (1) After enough data samples, the AFVINN algorithm and the traditional Q-learning algorithm converge to the same value; (2) the learning efficiency of the AFVINN algorithm is much higher than that of the traditional Q-learning algorithm. It is about 1000 times that of the traditional Q-learning algorithm. It can be seen that the AFVINN algorithm proposed by the present invention can not only transform high-dimensional control problems into one-dimensional control problems and simplify the solution process, but also keep the long-term average profits of MTO enterprises at a high level.

Table 2 shows that: (1) the AFVINN algorithm is better than the FCFS method in maximizing the long-term average profit of MTO companies; (2) when the order acceptance rate is lower than the FCFS method, the AFVINN algorithm can still maintain a higher average profit. It can be seen that the AFVINN algorithm can accept orders with higher profits with a higher probability in order acceptance strategy to achieve the purpose of maximizing the long-term average profit of the enterprise.

Table 2 Basic Scenario

AFVINN algorithm FCFS method Average profit: 367.2365 Average profit: 309.4537 Order acceptance rate: 0.1324 Order acceptance rate: 0.2698

Production capacity is very critical to the profitability of MTO companies. By changing the unit production capacity of the MTO enterprise and keeping other parameters the same as the basic scenario, we observe the changes in the order acceptance strategy of the MTO enterprise between the AFVINN algorithm and the FCFS method.

It can be seen from Figure 5 that (1) the AFVINN algorithm can always maintain a high profit level when facing different production capacities of enterprises; (2) when reducing the unit production capacity of enterprises, the AFVINN algorithm and the FCFS method respectively reduce the average profit. The average profits of the AFVINN algorithm and the FCFS method increased by 128.6104% and 122.9773% respectively when the unit production capacity increased from 20 to 35. It can be seen that the AFVINN algorithm can rationally utilize the limited resources of the enterprise, thereby creating higher profits for the enterprise and having better adaptability in the face of limited resources.

Order arrival rate is also an important factor in MTO companies' order acceptance decisions. By changing the order arrival rate and keeping other parameters the same as the basic scenario, we can observe the AFVINN algorithm (in Figures 6 and 7, in the first column of each group of graphs respectively) and the FCFS method (in Figures 6 and 7, respectively in each group of graphs). (Last column of group graph) Changes in MTO Enterprise Order Acceptance Strategies. It can be seen from Figure 6 and Figure 7: (1) When λ decreases, the order acceptance rate will increase. When λ increases, the order acceptance rate will decrease; this is because when the number of orders arriving per unit time increases, that is, two orders When the time interval between arrivals decreases, this will reduce the time for MTO companies to arrange accepted orders. Therefore, within the latest delivery deadline, the completion probability of accepted orders will decrease, resulting in a lower order acceptance rate. (2) Although the order acceptance rate under the AFVINN algorithm is lower than that of the FCFS method, the average profit is higher than the average profit of the FCFS method. It can be seen that the AFVINN algorithm can better adapt to the uncertainty of customer order arrival.

5.2 Model comparison

Existing literature [15-16] did not consider inventory costs when using reinforcement learning algorithms to model and solve order acceptance problems. This section will compare the order acceptance strategy of AFVINN algorithm with and without considering inventory cost. The former considers inventory cost in the process of modeling and solving the order acceptance problem; the latter considers the inventory cost in the process of modeling and solving the order acceptance problem. Inventory costs are not considered. It can be seen from Figures 8 and 9 that: (1) the order acceptance rate without considering inventory costs is higher than the order acceptance rate with inventory costs being considered, but the average revenue considering inventory cost factors is always higher than the average revenue without considering inventory cost factors; ( 2) When other factors remain unchanged, the order acceptance strategy of an enterprise that takes inventory cost into consideration will change with changes in inventory cost, while the order acceptance strategy of an enterprise that does not consider inventory cost will not be affected by changes in inventory cost. ; (3) When inventory costs continue to increase, the average profit decline trend of companies that consider inventory costs is slower than the average profit decline trend of companies that do not consider inventory costs. Therefore, in the process of modeling and solving the order acceptance problem of MTO enterprises, the factor of inventory cost is taken into consideration. The enterprise can make different order acceptance decisions based on different inventory costs to ensure that the enterprise's long-term average profit is maximized; and In real life, the existence of inventory costs often affects corporate profits, occupies corporate funds, and affects the operation of corporate funds. Therefore, the factor of inventory costs cannot be ignored during the order acceptance process.

Most of the existing literature only considers order characteristics and assumes that customers are equally important. Although literature [16] uses reinforcement learning ideas to model and solve customer priority factors, it only divides customer priorities into three Level, there are many kinds of customer priorities in real life. Therefore, the present invention sets the customer priority as an equal probability distribution on μ∈(0,1], in which the delay penalty cost and rejection cost are directly related to the customer priority. In this section of the experiment, the basic scenario is used as the benchmark , firstly, change the unit delay penalty cost under the condition that the rejection cost remains unchanged; secondly, change the rejection cost under the condition that the unit delay penalty cost remains unchanged.

Table 3 Changing the unit delay penalty cost and rejection cost

It can be seen from Figure 10 and Table 3: (1) The long-term average revenue of an enterprise that considers multiple customer priorities is greater than the average revenue that considers three customer priorities and does not consider customer priorities; (2) The customer level based on the AFVINN algorithm is greater than Or equal to 0.5, the order acceptance rate decreases with the increase of delay penalty cost; while the order acceptance rate with customer level less than 0.5 increases with the increase of delay penalty cost; (3) When the rejection cost increases, that is, the rejection cost has a negative impact on the MTO enterprise When the profit impact becomes greater and greater, when the AFVINN algorithm makes order acceptance decisions, the order acceptance rate for customer levels greater than or equal to 0.5 shows an upward trend, while the order acceptance rate for customer levels less than 0.5 shows a downward trend.

It can be seen from this that when the delay penalty cost is large, when accepting orders with higher customer priority, if the company does not complete production within the specified time limit, it will need to pay higher costs, so the acceptance rate of high-priority customer orders As the delay penalty cost increases, it decreases; when the rejection cost is high, the enterprise needs to bear higher costs for rejecting orders from high-priority customers, so the acceptance rate of orders from high-priority customers increases as the rejection cost increases. Therefore, when the delay penalty cost is large, the company can increase the number of orders accepted by customers with lower priority, and appropriately reduce the orders of high priority customers; when the cost of rejection is large, the company can increase the number of orders accepted by high-priority customers. Order. Therefore, when faced with different delay penalty costs and rejection costs, the AFVINN algorithm can promptly adjust the order acceptance strategy to minimize the impact of rejection costs on the average profits of MTO companies, so as to maximize the long-term average profits of the companies.

Based on the factors considered in traditional MTO enterprise order acceptance problems, this invention adds order inventory costs and various customer priority factors, constructs a Markov decision process order acceptance model, and uses the AFVINN algorithm to solve it. This algorithm can not only Converting the multi-dimensional state space in the MTO enterprise order acceptance problem into a one-dimensional state space simplifies the solution process and can keep the long-term average profit of the enterprise at a high level.

Simulation experiments show that in the MTO enterprise order acceptance problem, customer priority and inventory cost factors are important to the enterprise's order acceptance strategy and profit; compared with the traditional Q-learning algorithm, the AFVINN algorithm can solve high-dimensional control problems Convert it into a one-dimensional control problem, improve the efficiency of sample utilization, and simplify the solution process; the AFVINN-based algorithm is better than the FCFS-based method in maximizing the long-term average profit of the enterprise. This algorithm has a higher order acceptance and selection ability, and is more sensitive to environmental changes. Have good adaptability and be able to weigh order profits and various cost factors to bring higher profits to MTO companies. The present invention realizes the modeling of the arrival of dynamically changing orders (the information of the orders cannot be obtained in advance, and there is uncertainty in the current arriving orders), which is more consistent with the status of actual orders in reality. Modeling and solving take into account more comprehensive factors, such as: considering inventory cost factors, considering customer priorities, etc.; reducing the dimension of the state space reduces the computational difficulty when solving the model.

It is understood that the specific order or hierarchy of steps in the disclosed processes is an example of an exemplary approach. Based on design preferences, it is understood that the specific order or hierarchy of steps in the process may be rearranged without departing from the scope of the present disclosure. The accompanying method claims present elements of the various steps in a sample order, and are not intended to be limited to the specific order or hierarchy described.

In the foregoing detailed description, various features are grouped together in single embodiments to simplify the disclosure. This method of disclosure is not to be interpreted as reflecting an intention that embodiments of the claimed subject matter require more features than are expressly recited in each claim. Rather, as the following claims reflect, this invention lies in less than all features of a single disclosed embodiment. Thus, the following claims are hereby expressly incorporated into the Detailed Description, with each claim standing on its own as a separate preferred embodiment of this invention.

The disclosed embodiments are described above to enable any person skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit and scope of the disclosure. Therefore, this disclosure is not intended to be limited to the embodiments set forth herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

The above description includes examples of one or more embodiments. Of course, it is impossible to describe all possible combinations of components or methods for describing the above embodiments, but those of ordinary skill in the art will recognize that the various embodiments can be further combined and arranged. Accordingly, the embodiments described herein are intended to embrace all such alterations, modifications and variations that fall within the scope of the appended claims. Furthermore, to the extent that the term "comprises" is used in the description or claims, the word is encompassed in a manner similar to the term "includes," as if "comprises," is interpreted as a connective in the claims. Furthermore, any term "or" used in the description of the claims is intended to mean "a non-exclusive or".

The above-described specific embodiments further describe the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above-mentioned are only specific embodiments of the present invention and are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included in the protection scope of the present invention.

Claims (6)

1. An MTO enterprise order processing method, which is characterized by including: When a current order arrives at an order-oriented production MTO enterprise, an order acceptance strategy model based on the Markov decision process MDP theory is established based on the current order queue of the MTO enterprise and the currently arriving orders. The order based on the MDP theory The acceptance strategy model is used to determine the optimal strategy; among them, the current arrival order represents the order that the MTO enterprise has received but has not yet decided whether to accept; the strategy is to accept the current arrival order or reject the current arrival order, and the optimal strategy refers to When choosing this strategy, the interests of the MTO company are optimal; The order acceptance strategy model based on the MDP theory is transformed according to the post-state theory of the reinforcement learning algorithm, and the MDP model based on the post-state is obtained after the transformation; the solution of the MDP model based on the post-state is reduced through the learning process of the learning parameter vector of the reinforcement learning algorithm. Difficulty, obtain the MDP optimization model based on the post-state; A three-layer artificial neural network is used to solve the post-state-based MDP optimization model to obtain the solution results. Based on the solution results, the optimal strategy for accepting the current arriving order is determined; When a current order arrives at an order-oriented production MTO enterprise, an order acceptance strategy model based on the Markov decision process MDP theory is established based on the current order queue and currently arriving orders of the MTO enterprise, specifically including: Assume that each order is not split into production, that the order is shipped to the customer in one go after production is completed, and that once the MTO company accepts the order, the order cannot be changed or canceled; Determine the information of each order in the current order queue and the current arriving order based on the current order queue and the current arriving order. The order information includes: the customer priority μ corresponding to the order, the unit product price pr, the required product quantity q, the lead time lt and the minimum Late delivery date dt; among them, the order in the current order queue is an accepted order, the order arrival obeys the Poisson distribution with parameter λ, and the unit product price and product demand quantity in the order obey uniform distribution respectively; Determine the reward sub-items faced based on the information of each order in the current order queue and the current arriving order. The reported sub-items include: rejection cost of rejecting the current arriving order, profit of accepting the currently arriving order, and delay penalty for orders in the current order queue. Cost, the inventory cost of the order in the current order queue; where: If the currently arriving order is rejected, a rejection cost will be incurred: μ*J, where J represents the rejection cost without considering customer priority; If the current arrival order is accepted, the profit of the order I will be obtained: I=pr*q, and the production cost C will be consumed at the same time: C=c*q, where c is the unit product production cost; For each order in the current order queue, the MTO company will produce according to the principle of first come, first served. If there is a delayed order in the current order queue, and the delivery time of the delayed order is within the latest delivery period, the MTO company will The delay penalty cost Y is paid to the customer corresponding to the delayed order: Among them, t represents the production time still required for the accepted order, b represents the unit production capacity of the MTO enterprise, and u represents the unit product delay penalty cost per unit time of the MTO enterprise; If the product of the order in the current order queue is generated within the lead time and the product is temporarily stored in the warehouse, the product is temporarily stored in the MTO enterprise warehouse, resulting in inventory cost N: Among them, h represents the inventory cost per unit product per unit time; According to the Markov decision process MDP theory, as well as the information and return items of each order in the current order queue and the current arriving order, an order acceptance strategy model based on the MDP theory of the MTO enterprise is established. The MTO enterprise's order acceptance strategy model is based on the MDP theory. The order acceptance strategy model is a four-tuple (S, A, f, R), which are the state space S, the action space A, the state transition function f, and the reward function R, where: The state space S represents the state of the system where the MTO enterprise order processing method is located; the state space S is an nx6-dimensional vector, where n represents the number of order types, and 6 represents 6 types of order information: customer priority μ, unit product price pr, Product demand quantity q, lead time lt, latest delivery date dt still requires production completion time t for orders in the current order queue, where t has a preset maximum upper limit value; Action space A represents the action set for the currently arriving order; when the current order arrives at time m, the MTO enterprise needs to make an action decision of accepting the order or rejecting the order, and the action set of accepting the order or rejecting the order forms the action space amount A, A=(a ₁ ,a ₂ ), where a ₁ means accepting the order and a ₂ means rejecting the order; The state transition function f represents the transition from the current state to the m decision time state, and the m decision time refers to the time when action is taken for the current arriving order; the generation process of the state transition function f is: Assume that the order information μ, pr, q, lt, dt are all independent and identically distributed. According to the initial state s and the action a taken at decision time m, the probability density function f of the state at the next decision time m+1 is (·|(s,a)) are respectively expressed as f _M (x), f _PR (x), f _Q (x), f _LT (x), f _DT (x); And the order information μ _m+1 , pr _m+1 , q _m+1 , lt _m+1 , dt _m+1 at the next decision time m+1 is independent with respect to (s _m , q _m ); where, t _m+1 is expressed as: Equation (1) indicates that t _m+1 is affected by (s _m , a _m ), and different (q _m , t _m , a _m ) lead to different order production times, and t _m+1 is also affected by the order arrival time. The influence of interval; among them, AT _m→m+1 represents the time interval between the arrival of two orders, that is, according to the arrival of each order, it obeys the Poisson distribution with parameter λ; According to the characteristics of the current state s, action a, and order information μ _m+1 , pr _m+1 , q _m+1 , lt _m+1 , dt _m+1 with respect to (s _m , a _m ), we get the following The conditional probability density of state s′ at a decision moment m+1, and the conditional probability density of state s′ at m+1 at the next decision moment are expressed as: f(s′|s,a)＝f _M (μ′)*f _PR (pr′)*f _Q (q′)*f _LT (lt′)*f _DT (dt′)*f _T (t′ |s,a), where f _T (r′|s,a) represents the production time still required to produce the accepted order at the next decision moment m+1 after taking action a in the current state s, f _T (t′|s,a ) is defined by formula (1) and related random variables; When the MTO enterprise makes a decision at m, the reward function R represents the corresponding reward obtained after taking actions on the currently arrived order. The reward function R is expressed as: Among them, when a _m =1 means that if the MTO enterprise receives the current arriving order, the reward function R is expressed as ICYN; When a _m = 0, it means that if the MTO enterprise rejects the current arriving order, the reward function R is expressed as -μ*J; For any strategy in the order acceptance strategy model based on MDP theory, its corresponding value function is defined according to the reward function, and the average long-term profit corresponding to the strategy is represented by the value function. The value function is expressed as: Among them, π represents any strategy, γ represents future reward discount, 0<γ≤1, by setting γ to ensure that the summation term defined by the formula is meaningful, n represents the total number of decision-making moments, and each current order corresponds to a decision-making moment; The optimal strategy π ^* is determined by the average long-term profit of any strategy π. The optimal strategy π ^* is used to ensure the long-term profit maximization of the enterprise. At this time, the interests of the MTO enterprise are optimal; the optimal strategy π ^* is expressed as: Among them, Π represents the set of all strategies; The order acceptance strategy model based on the MDP theory is transformed according to the post-state theory of the reinforcement learning algorithm. After the transformation, an MDP model based on the post-state is obtained, which specifically includes: After receiving the current arriving order and making a decision at time m, set the post-state variable p _m according to the reinforcement learning algorithm. The post-state variable represents the production time still needed to produce the accepted order after selecting action a _m at decision time m; where, The post-state is an intermediate variable between two consecutive states; The post-state variable p _m is determined according to the current state s _m and action a _m , and the post-state variable p _m is expressed as: According to p _m, the production time t m+1 still required for the order that has been accepted at the next decision time _m+1 is expressed as: Among them, in formula (5), Indicates the larger number between the variables ,dt′,t′) conditional probability density is expressed as: Among them, the conditional probability density function f _T (·|p) is defined by equation (5) and related random variables; After setting the post-state variable, rewrite the conditional expectation E[] in the MDP theory, and after rewriting the conditional expectation E[], define the value function of the post-state, and express the value function of the post-state as: J ^* (p)＝γE[V ^* (s′)|p] (7) The optimal strategy π ^* is constructed through the post-state value function, thereby turning the optimal strategy π ^* into a one-dimensional state space. The optimal strategy π ^* is expressed as: 2. The MTO enterprise order processing method according to claim 1, characterized in that the learning process of the learning parameter vector through the reinforcement learning algorithm reduces the difficulty of solving the MDP model based on the post-state, and obtains the MDP optimization based on the post-state. Models, specifically including: The reinforcement algorithm does not directly calculate J ^* when constructing the optimal policy ^{π * through} the post-state value function J * for solving, but adopts the learning process of learning parameter vectors to realize the approximation of J ^* for solving; adopts the method of learning parameter vectors The learning process is used to achieve the approximate solution of J ^* , as follows: Determined according to the given parameter vector θ as well as The parameter vector θ is used to perform parameter learning from data samples using the reinforcement algorithm to obtain the parameter vector θ ^* , and the parameter vector θ ^* obtained through learning is determined To approximate J ^* , determine the optimal strategy π ^* based on the approximate J ^* . 3. The MTO enterprise order processing method according to claim 2, characterized in that the three-layer artificial neural network is used to solve the MDP optimization model based on the post-state to obtain the solution result, and it is determined according to the solution result whether the current arrival order is The optimal strategy accepted includes: J ^* (p) is solved through a three-layer artificial neural network ANN, and J ^* (p) is approximated to arbitrary precision. Using a three-layer artificial neural network ANN is expressed as: Among them, the parameter vector can be expressed as: θ=[w ₁ ,...,w _N ,α ₁ ,...α _N ,u ₁ ,...u _N ,β], Φ _H (x)＝1/(1+e ^-x ) Function of equation (11) It is a three-layer single-input single-output neural network. It has only a single-node input layer, whose output represents the value of the post-state p, and a hidden layer containing N nodes. The input of the i-th node is the weighted post-state. The sum of the value w _i *p and the hidden layer bias α _i ; and the input-output relationship of each hidden layer node is represented by the Φ _H (·) function, where Φ _H (·) is called the activation function; the output layer There is a node. The input of the output layer is the sum of the weighted output of the hidden layer and the bias β of the output layer. The output of the output layer represents the final approximate function value/> 4. An MTO enterprise order processing system, which is characterized by including: The model building unit is used to establish an order acceptance strategy model based on the Markov decision process MDP theory based on the current order queue of the MTO enterprise and the currently arriving orders when the current order arrives at the order-oriented production MTO enterprise. The above-mentioned order acceptance strategy model based on MDP theory is used to determine the optimal strategy; among them, the current arrival order represents the order that the MTO enterprise has received but has not yet decided whether to accept; the strategy is to accept the current arrival order or reject the current arrival order, so The above-mentioned optimal strategy means that the interests of the MTO enterprise are optimal when choosing this strategy; The model transformation unit is used to transform the order acceptance strategy model based on the MDP theory according to the post-state theory of the reinforcement learning algorithm. After the transformation, an MDP model based on the post-state is obtained; The model optimization unit is used to reduce the difficulty of solving the post-state-based MDP model through the learning process of the learning parameter vector of the reinforcement learning algorithm, and obtain the post-state-based MDP optimization model; The solving unit is used to use a three-layer artificial neural network to solve the post-state-based MDP optimization model to obtain the solution results, and determine the optimal strategy for accepting the current arriving order based on the solution results; The model building unit is specifically used for: Assume that each order is not split into production, that the order is shipped to the customer in one go after production is completed, and that once the MTO company accepts the order, the order cannot be changed or canceled; Determine the information of each order in the current order queue and the current arriving order based on the current order queue and the current arriving order. The order information includes: the customer priority μ corresponding to the order, the unit product price pr, the required product quantity q, the lead time lt and the minimum Late delivery date dt; among them, the order in the current order queue is an accepted order, the order arrival obeys the Poisson distribution with parameter λ, and the unit product price and product demand quantity in the order obey uniform distribution respectively; Determine the reward sub-items faced based on the information of each order in the current order queue and the current arriving order. The reported sub-items include: rejection cost of rejecting the current arriving order, profit of accepting the currently arriving order, and delay penalty for orders in the current order queue. Cost, the inventory cost of the order in the current order queue; where: If the currently arriving order is rejected, a rejection cost will be incurred: μ*J, where J represents the rejection cost without considering customer priority; If the current arrival order is accepted, the profit of the order I will be obtained: I=pr*q, and the production cost C will be consumed at the same time: C=c*q, where c is the unit product production cost; For each order in the current order queue, the MTO company will produce according to the principle of first come, first served. If there is a delayed order in the current order queue, and the delivery time of the delayed order is within the latest delivery period, the MTO company will The delay penalty cost Y is paid to the customer corresponding to the delayed order: Among them, t represents the production time still required for the accepted order, b represents the unit production capacity of the MTO enterprise, and u represents the unit product delay penalty cost per unit time of the MTO enterprise; If the product of the order in the current order queue is generated within the lead time and the product is temporarily stored in the warehouse, the product is temporarily stored in the MTO enterprise warehouse, resulting in inventory cost N: Among them, h represents the inventory cost per unit product per unit time; According to the Markov decision process MDP theory, as well as the information and return items of each order in the current order queue and the current arriving order, an order acceptance strategy model based on the MDP theory of the MTO enterprise is established. The MTO enterprise's order acceptance strategy model is based on the MDP theory. The order acceptance strategy model is a four-tuple (S, A, f, R), which are the state space S, the action space A, the state transition function f, and the reward function R, where: The state space S represents the state of the system where the MTO enterprise order processing method is located; the state space S is an nx6-dimensional vector, where n represents the number of order types, and 6 represents 6 types of order information: customer priority μ, unit product price pr, Product demand quantity q, lead time lt, latest delivery date dt still requires production completion time t for orders in the current order queue, where t has a preset maximum upper limit value; Action space A represents the action set for the currently arriving order; when the current order arrives at time m, the MTO enterprise needs to make an action decision of accepting the order or rejecting the order, and the action set of accepting the order or rejecting the order forms the action space amount A, A=(a ₁ ,a ₂ ), where a ₁ means accepting the order and a ₂ means rejecting the order; The state transition function f represents the transition from the current state to the m decision time state, and the m decision time refers to the time when action is taken for the current arriving order; the generation process of the state transition function f is: Assume that the order information μ, pr, q, lt, dt are all independent and identically distributed. According to the initial state s and the action a taken at decision time m, the probability density function f of the state at the next decision time m+1 is (·|(s,a)) are respectively expressed as f _M (x), f _PR (x), f _Q (x), f _LT (x), f _DT (x); And the order information μ _m+1 , pr _m+1 , q _m+1 , lt _m+1 , dt _m+1 at the next decision time m+1 is independent with respect to (s _m , a _m ); where, t _m+1 is expressed as: Equation (1) indicates that t _m+1 is affected by (s _m , a _m ), and different (q _m , t _m , a _m ) lead to different order production times, and t _m+1 is also affected by the order arrival time. The influence of interval; among them, AT _m→m+1 represents the time interval between the arrival of two orders, that is, according to the arrival of each order, it obeys the Poisson distribution with parameter λ; According to the characteristics of the current state s, action a, and order information μ _m+1 , pr _m+1 , q _m+1 , lt _m+1 , dt _m+1 with respect to (s _m , a _m ), we get the following The conditional probability density of state s′ at a decision moment m+1, and the conditional probability density of state s′ at m+1 at the next decision moment are expressed as: f(s′|s,a)＝f _M (μ′)*f _PR (pr′)*f _Q (q′)*f _LT (lt′)*f _DT (dt′)*f _T (t′ |s,a) where f _T (t′|s,a) represents the production time still required to produce the accepted order at the next decision moment m+1 after taking action a in the current state s, f _T (t′|s,a ) is defined by formula (1) and related random variables; When the MTO enterprise makes a decision at m, the reward function R represents the corresponding reward obtained after taking actions on the currently arrived order. The reward function R is expressed as: Among them, when a _m =1 means that if the MTO enterprise receives the current arriving order, the reward function R is expressed as ICYN; When a _m = 0, it means that if the MTO enterprise rejects the current arriving order, the reward function R is expressed as -μ*J; For any strategy in the order acceptance strategy model based on MDP theory, its corresponding value function is defined according to the reward function, and the average long-term profit corresponding to the strategy is represented by the value function. The value function is expressed as: Among them, π represents any strategy, γ represents future reward discount, 0<γ≤1, by setting γ to ensure that the summation term defined by the formula is meaningful, n represents the total number of decision-making moments, and each current order corresponds to a decision-making moment; The optimal strategy π ^* is determined by the average long-term profit of any strategy π. The optimal strategy π ^* is used to ensure the long-term profit maximization of the enterprise. At this time, the interests of the MTO enterprise are optimal; the optimal strategy π ^* is expressed as: Among them, Π represents the set of all strategies; The model conversion unit is specifically used for: After receiving the current arriving order at time m and making a decision, the post-state variable p _m is set according to the reinforcement learning algorithm. The post-state variable represents the production time still needed to produce the accepted order after selecting action a m at decision time _m ; where, The post-state is an intermediate variable between two consecutive states; The post-state variable p _m is determined according to the current state s _m and action a _m , and the post-state variable p _m is expressed as: According to p _m, the production time t m+1 still required for the order that has been accepted at the next decision time _m+1 is expressed as: Among them, in formula (5), Indicates the larger number between the variables ,dt′,t′) conditional probability density is expressed as: Among them, the conditional probability density function f _T (·|p) is defined by equation (5) and related random variables; After setting the post-state variable, rewrite the conditional expectation E[] in the MDP theory, and after rewriting the conditional expectation E[], define the value function of the post-state, and express the value function of the post-state as: J ^* (p)＝γE[V ^* (s′)|p] (7) The optimal strategy π ^* is constructed through the post-state value function, thereby turning the optimal strategy π ^* into a one-dimensional state space. The optimal strategy π ^* is expressed as: 5. The MTO enterprise order processing system according to claim 4, characterized in that the model optimization unit is specifically used for: The reinforcement algorithm does not directly calculate J ^* when constructing the optimal policy ^{π * through} the post-state value function J * for solving, but adopts the learning process of learning parameter vectors to realize the approximation of J ^* for solving; adopts the method of learning parameter vectors The learning process is used to achieve the approximate solution of J ^* , as follows: Determined according to the given parameter vector θ as well as The parameter vector θ is used to perform parameter learning from data samples using the reinforcement algorithm to obtain the parameter vector θ ^* , and the parameter vector θ ^* obtained through learning is determined To approximate J ^* , determine the optimal strategy π ^* based on the approximate J ^* . 6. The MTO enterprise order processing system according to claim 5, characterized in that the solving unit is specifically used for: J ^* (p) is solved through a three-layer artificial neural network ANN, and J ^* (p) is approximated to arbitrary precision. Using a three-layer artificial neural network ANN is expressed as: Among them, the parameter vector can be expressed as: θ=[w ₁ ,...,w _N ,α ₁ ,...α _N ,u ₁ ,...u _N ,β] Φ _H (x)＝1/(1+e ^-x ) Function of equation (11) It is a three-layer single-input single-output neural network. It has only a single-node input layer, whose output represents the value of the post-state p, and a hidden layer containing N nodes. The input of the i-th node is the weighted post-state. The sum of the value w _i *p and the hidden layer bias α _i ; and the input-output relationship of each hidden layer node is represented by the Φ _H (·) function, where Φ _H (·) is called the activation function; the output layer There is a node. The input of the output layer is the sum of the weighted output of the hidden layer and the output layer bias β. The output of the output layer represents the final approximate function value/>