JP7287414B2 - Logistics rule learning device, logistics rule execution device, logistics rule learning method, and logistics rule learning program - Google Patents

Description

The present invention relates to a physical distribution rule learning device, a physical distribution rule execution device, a physical distribution rule learning method, and a physical distribution rule learning program for creating physical distribution rules for a plurality of transportation equipment for transporting objects to be transported.

In Patent Document 1, a learning data reception unit that receives learning data including learning purposes and allowable requirements for learning operations performed when learning control, a neural network that performs learning based on the learning data, and a neural network and an output unit for outputting a learning result by, the neural network performing a first learning to achieve an initial stage of the learning objective, and based on the result of the first learning, the learning behavior meets the allowable requirements. A learning device is described that performs second learning to learn a control range, and performs third learning to achieve a learning objective within the control range based on the result of the second learning.

JP 2018-200539 A

In the learning device described in Patent Literature 1, when learning a distribution rule for a plurality of transportation equipment that involves interference between the transportation equipment, it is possible to learn the control range of the transportation equipment that can avoid the interference between the transportation equipment. It is difficult, and it is not possible to avoid interference between transportation equipment only by learning the control range. In addition, the physical distribution rules for a plurality of transportation equipment that involve interference between transportation equipment become very complicated if described in an if-then format, etc., and management such as maintenance is not easy.

The present invention has been made in view of the above problems, and aims to provide a physical distribution rule learning device, a physical distribution rule execution device, and a physical distribution rule learning device capable of easily creating a physical distribution rule for a plurality of transportation devices that involve interference between the transportation devices. An object of the present invention is to provide a physical distribution rule learning method and a physical distribution rule learning program.

A physical distribution rule learning device according to the present invention is a physical distribution rule learning device for creating physical distribution rules for a plurality of transportation equipment for transporting objects to be transported, and includes a simulator section for reproducing a physical distribution environment including movement of the transportation equipment. a learning-time next-action determining unit that determines a next action of the transportation equipment in the physical distribution environment and instructs the transportation equipment; a learning-time state interpreting unit for interpreting the obtained logistics environment state; and a reward for calculating a reward value for evaluating the next action of the transportation equipment based on the logistics environment state interpreted by the learning-time state interpreting unit. a calculation unit, the next action of the transportation equipment instructed by the next action determination unit during learning, the state of the logistics environment interpreted by the state interpretation unit during learning, and the reward value calculated by the reward calculation unit; and a learning unit that corrects and learns the physical distribution rule.

The physical distribution rule learning apparatus according to the present invention is characterized in that, in the above invention, the simulator section is a discrete event simulator that reproduces the physical distribution environment including the movement of the transportation equipment.

A physical distribution rule learning apparatus according to the present invention is characterized in that, in the above invention, the state of the physical distribution environment interpreted by the state interpreting section at the time of learning is the state of the physical distribution environment when the discrete event occurs.

In the physical distribution rule learning device according to the present invention, in the above invention, the state of the physical distribution environment at the time of occurrence of the discrete event includes the destination including the position and stop state of the transportation equipment, the presence or absence of the transportation object loaded on the transportation equipment. and destination, and the location and destination of the transport object to be loaded onto the transportation equipment.

In the physical distribution rule learning device according to the present invention, in the above invention, the reward calculation unit includes a value for correctly transporting the transport object by the transport equipment, a value for transport time within one unit of learning, and a value for transport time within one learning unit. The reward value is calculated based on at least one of the values relating to transport failure within the unit.

The physical distribution rule learning device according to the present invention is characterized in that, in the above invention, the learning unit performs learning using a neural network.

The physical distribution rule learning apparatus according to the present invention is characterized in that, in the above invention, the learning unit performs learning using the same neural network for a plurality of transport equipment whose movements are symmetrical with each other.

A physical distribution rule execution device according to the present invention includes a learning result storage unit for storing learning results of the learning unit included in the physical distribution rule learning device according to the present invention, and a learning result reading unit for reading the learning results from the learning result storage unit. a runtime state interpretation unit that interprets the state of the physical distribution environment in the real space or the state of the physical distribution environment reproduced by the second simulator; and an execution-time next-action determining unit that determines the next action of the transportation equipment in the physical distribution environment using the state of the physical distribution environment interpreted by the state interpretation unit and instructs the transportation equipment.

A physical distribution rule execution apparatus according to the present invention is characterized in that, in the above invention, the state of the physical distribution environment interpreted by the runtime state interpretation unit is the state of the physical distribution environment when the discrete event occurs.

In the physical distribution rule execution apparatus according to the present invention, in the above invention, the state of the physical distribution environment at the time of occurrence of the discrete event includes the destination including the position and stop state of the transportation equipment, the presence or absence of the transportation object loaded on the transportation equipment. and destination, and the location and destination of the transport object to be loaded onto the transportation equipment.

A physical distribution rule learning method according to the present invention is a physical distribution rule learning method for creating physical distribution rules for a plurality of transportation equipment for transporting objects to be transported, and is a first step of reproducing a physical distribution environment including movement of the transportation equipment. and a second step of determining the next action of the transportation equipment in the physical distribution environment and instructing the transportation equipment, and reproducing the instructed next behavior of the transportation equipment in the first step. a third step of interpreting the state of the logistics environment to be processed; a fourth step of calculating a reward value for evaluating the next action of the vehicle based on the state of the logistics environment interpreted in the third step; The next action of the transportation equipment instructed in the second step, the state of the physical distribution environment interpreted in the third step, and the remuneration value calculated in the fourth step are used as inputs to correct and learn the physical distribution rule. and five steps.

A physical distribution rule learning program according to the present invention is a physical distribution rule learning program that causes a computer to execute processing for creating physical distribution rules for a plurality of transportation equipment for transporting objects to be transported, and is a physical distribution environment including movement of the transportation equipment. a second process of determining the next action of the transportation equipment in the logistics environment and instructing the transportation equipment, and the first process of determining the next action instructed to the transportation equipment and calculating a reward value for evaluating the next action of the transportation equipment based on the state of the physical distribution environment interpreted in the third processing. a fourth process, the next action of the transportation equipment instructed in the second process, the state of the physical distribution environment interpreted in the third process, and the remuneration value calculated in the fourth process as inputs; and a fifth process of correcting and learning by a computer.

According to the physical distribution rule learning device, the physical distribution rule execution device, the physical distribution rule learning method, and the physical distribution rule learning program according to the present invention, it is possible to easily create physical distribution rules for a plurality of transportation equipment that involve interference between transportation equipment. .

FIG. 1 is a block diagram showing the configuration of a physical distribution control system that is one embodiment of the present invention. FIG. 2 is a flow chart showing the flow of distribution rule learning processing, which is one embodiment of the present invention. FIG. 3 is a diagram showing a configuration example of a simulation model. FIG. 4 is a diagram showing a configuration example of DQN. FIG. 5 is a flowchart showing the flow of model execution processing shown in FIG. FIG. 6 is a diagram showing an example in which the simulation model shown in FIG. 3 is made to learn a physical distribution rule for transporting 10 cargoes according to the present invention.

A physical distribution control system according to an embodiment of the present invention will be described below with reference to the drawings.

〔composition〕
First, referring to FIG. 1, the configuration of a physical distribution control system that is an embodiment of the present invention will be described.

FIG. 1 is a block diagram showing the configuration of a physical distribution control system that is one embodiment of the present invention. As shown in FIG. 1, a physical distribution control system 1, which is an embodiment of the present invention, comprises a learning device 2, a common device 3, and an execution device 4.

The learning device 2 is composed of a well-known information processing device such as a computer, and prepares physical distribution rules for a plurality of transportation equipment for transporting objects to be transported by learning processing. In the present embodiment, the learning apparatus 2 includes a simulator unit 21, a next action determination unit 22 during learning, a state interpretation unit 23 during learning, and a reward calculation unit 24 by executing a computer program by an arithmetic processing unit inside the information processing apparatus. , and the learning unit 25 .

The simulator unit 21 uses the information output from the learning-time-next-action determination unit 22 to reproduce, by simulation, the physical distribution environment of a plurality of transportation equipment executed in real space, including the movement of the transportation equipment.

The learning-time next action determination unit 22 determines the next action of the transportation equipment in the distribution environment reproduced by the simulator unit 21 and outputs information indicating the determined next action of the transportation equipment to the simulator unit 21 .

The learning-time state interpreting unit 23 interprets the event state of the logistics environment caused by the next action of the transportation equipment determined by the learning-time next action determining unit 22, and transmits information indicating the event state of the logistics environment to the reward calculation unit 24 and the learning. Output to unit 25 .

The reward calculation unit 24 calculates a reward value for evaluating the next action of the transportation equipment determined by the next action determination unit 22 during learning based on the event state of the logistics environment interpreted by the state interpretation unit 23 during learning. Information indicating the reward value obtained is output to the learning unit 25 .

The learning unit 25 learns the next action of the transportation equipment determined by the next action determination unit 22 when learning, the event state of the logistics environment interpreted by the state interpretation unit 23 when learning, and the reward value calculated by the reward calculation unit 24. By repeating the learning process by the neural network as data, a neural network is created as a distribution rule for a plurality of transportation equipment, in which the event state of the distribution environment is input and the value that determines the next action of the transportation equipment is output.

The common device 3 is connected to the learning device 2 and the execution device 4 and has a learning result storage unit 31 . The learning result storage unit 31 is configured by a non-volatile storage device, and stores information about the structure and weights of the neural network learned by the learning unit 25 as learning results.

The execution device 4 is composed of a well-known information processing device such as a computer, and uses the logistics rules for a plurality of transportation devices created by learning processing to determine the next action of the transportation device when transporting the object to be transported (at the time of execution). decide. In the present embodiment, the execution device 4 functions as a learning result reading unit 41, an execution state interpretation unit 42, and an execution next action determination unit 43 by the arithmetic processing unit inside the information processing apparatus executing a computer program. do.

The learning result reading unit 41 reads information about the structure and weights of the learned neural network from the learning result storage unit 31, and reconstructs the learned neural network (distribution rule).

The run-time state interpretation unit 42 interprets the event state of the distribution environment in the real space or the distribution environment reproduced by the second simulator unit having the same configuration as the simulator unit 21 (real space/distribution environment of the second simulator unit 5). and outputs the interpreted information indicating the event state of the physical distribution environment of the real space/second simulator unit 5 to the execution time next action determination unit 43 . The reason for arranging the second simulator unit here is to use it for the purpose of checking its operation and evaluating its performance before applying the learned distribution rule to the actual distribution environment. The second simulator section may use the simulator section 21 as it is, or may be configured separately as a second simulator section as shown here.

The run-time next action determination unit 43 inputs the event state of the physical space/distribution environment of the second simulator unit 5 interpreted by the run-time state interpretation unit 42 to the learned neural network, and outputs the learned neural network. determines the next action of the vehicle within the logistics environment based on For example, when the output of the learned neural network is the action evaluation function value (Q value), the execution-time next action determination unit 43 determines the action with the maximum Q value as the next action of the transportation equipment in the logistics environment. . Then, the execution-time next action determination unit 43 instructs the next action to the transportation equipment in the physical distribution environment of the real space/second simulator unit 5 based on the determination result.

The physical distribution control system executes the physical distribution rule learning process described below, thereby making it possible to easily create a physical distribution rule for a plurality of transportation equipments that involve interference between the transportation equipments. The operation of the physical distribution control system 1 when executing the physical distribution rule learning process, which is one embodiment of the present invention, will be described below with reference to FIG.

[Logistics rule learning process]
FIG. 2 is a flow chart showing the flow of distribution rule learning processing, which is one embodiment of the present invention. The flow chart shown in FIG. 2 starts at the timing when a command to execute the distribution rule learning process is input, and the distribution rule learning process proceeds to the process of step S1.

In the following, the distribution rule learning process will be described by taking as an example a case where the distribution rule for the distribution environment reproduced by the simulation model 6 shown in FIG. 3 is created by the learning process. The simulation model 6 shown in FIG. 3 uses #1, 2, and 3 cranes traveling on the same track (L01 to L11) to transport objects (hereinafter referred to as cargo) generated at generation locations P1 to P3. It is a discrete event type simulation model that delivers to the destinations D1-D4 specified in the cargo. In the physical distribution rule learning process of the present embodiment, physical distribution rules for cranes #1, 2, and 3 for correctly transporting all N loads are created, and the simulation model 6 operates on the following assumptions.

[cargo]
・Randomly generated from generation locations P1 to P3, and the generation interval follows a given probability distribution.
・Destinations D1 to D4 are also randomly determined.
・The cargo disappears when it is transported to destinations D1 to D4.

[crane]
・Run independently on the left and right on the track divided into 11 pieces.
- Each crane receives the next action and executes the received next action.
・The next action is the crane's destination track (L01 to L11), and the crane that receives the next action travels to the destination track.
• The crane can only load one load.
・The travel speed of the crane and the time required for loading and unloading are given in advance.

[Simulation end condition]
Terminate as successful if N shipments have been transported to the correct destinations.
・When two or more cargoes are stagnated at the generation locations P1 to P3, the operation is terminated as unsuccessful.
・If the cranes collide with each other, the operation is terminated as unsuccessful.

Moreover, in this embodiment, DQN (Deep Q-Network) is used as a learning algorithm. DQN enhances the expressiveness of the Q value by combining the expression of the action evaluation function value (Q value) of a reinforcement learning method called Q-learning with a deep neural network. FIG. 4 shows a configuration example of DQN. In the DQN shown in FIG. 4, the environment is the simulation model 6 shown in FIG. The agent determines the next action a(i, t) (i=1 to 3) of the #1, 2, and 3 cranes at time t and instructs the environment, and the event state s(t) of the environment and the reward value r Get (i, t). Then, the agent calculates the action evaluation function value Q(s, a) of the next action a (=a ₁ to a _M ) of the #1, 2, and 3 cranes that can be taken from the event state s(t) for each crane as follows: It is updated by the formula (1) shown. In formula (1), n indicates each agent. Learning in the deep neural network is performed by regarding the value shown in the following formula (2) as the correct value and adjusting the weight of each layer so that the value shown in the following formula (3) is minimized. A(s(t+1)) in equation (2) denotes a set of actions in state s(t+1), and D in equation (3) denotes a set of episodes representing a series of learning units.

Here, in formula (1), α (0<α<1) is a learning rate that determines the speed of learning, and γ (0<γ<1) is for decreasing the Q value along with the number of times of learning in order not to diverge the Q value. is the discount rate of When learning the physical distribution rule in chronological order, the chronological order of the next action is traced back to the past and the Q value is updated.

DQN not only selects the action that maximizes the reward value, but also executes the ε-greedy method that randomly selects actions with a certain probability in order to generate more efficient logistics rules. The ε-greedy method prepares an appropriate constant ε and generates a random number between 0 and 1 when selecting the next action. Choose actions with a high Q value.

In the process of step S1, the learning device 2 initializes the learning unit 25 and the number of episodes. Specifically, the learning device 2 randomly sets the weights of the neural network of the learning unit 25 . Note that an episode indicates a unit of learning, and in the present embodiment, one episode is learning for one execution of a physical distribution simulation. As a result, the process of step S1 is completed, and the distribution rule learning process proceeds to the process of step S2.

In the process of step S2, the learning device 2 increments the number of episodes by one. As a result, the process of step S2 is completed, and the distribution rule learning process proceeds to the process of step S3.

In the processing of step S3, the learning device 2 determines whether or not the number of episodes is greater than a predetermined number M. As a result of determination, if the number of episodes is greater than the predetermined number M (step S3: Yes), the learning device 2 advances the distribution rule learning process to the process of step S4. On the other hand, when the number of episodes is equal to or less than the predetermined number M (step S3: No), the learning device 2 advances the logistics rule learning process to the process of step S5.

In the process of step S4, the learning device 2 stores information about the structure and weights of the neural network learned by the learning section 25 in the learning result storage section 31 of the common device 3 as a learning result. As a result, the process of step S4 is completed, and the series of logistics rule learning process ends.

In the process of step S5, the learning device 2 initializes the simulation model 6 shown in FIG. Specifically, the learning device 2 determines the positions of the #1, 2, and 3 cranes in the simulation model 6 shown in FIG. Set the travel speed and the time required for loading and unloading cargo. As a result, the process of step S5 is completed, and the distribution rule learning process proceeds to the process of step S6.

In the process of step S6, the learning-time next action determining unit 22 determines the next action a(i, t) of the crane to be processed (corresponding crane) in the simulation model 6 shown in FIG. The next action a(i, t) of the corresponding crane is instructed to the unit 21 . As a result, the process of step S6 is completed, and the distribution rule learning process proceeds to the process of step S7.

In the process of step S7, the simulator unit 21 reproduces the physical distribution environment by executing the simulation model 6 based on the next action a(i, t) of the corresponding crane instructed by the learning next action determination unit 22, A state table indicating the state of the physical distribution environment shown in Tables 1 to 3 below is created (model execution processing). The details of this model execution processing will be described later with reference to the flowchart shown in FIG. As a result, the process of step S7 is completed, and the distribution rule learning process proceeds to the process of step S8.

In the process of step S8, the learning state interpretation unit 23 and the reward calculation unit 24 use the state table created by the process of step S7 to determine the physical distribution environment corresponding to the next action a(i, t) of the corresponding crane. Compute the event state s(t) and the reward value r(i,t). In the simulation model 6 shown in FIG. 3, since the #1 crane 1 and the #3 crane move in line symmetry with each other, only one neural network may be used to learn the physical distribution rules of the #1 crane and the #3 crane. . The inputs of the neural network are 18 values generated by the state interpretation unit 23 during learning, and the outputs are 11 Q values relating to the destination of the crane, which is the next action of the crane. Table 4 shows an input example of the neural network for the #1 and #3 cranes, Table 5 shows an input example of the neural network for the #2 crane, and Table 6 shows an output example of the neural network. Further, in this embodiment, the reward value is given within the range of -1.0 to 1.0, and the values shown in Table 7 are calculated from the state table shown in Table 1.

In the process of step S9, the learning unit 25 updates the formula (1) based on the value of the event state s(t) of the distribution environment and the reward value r(i, t) calculated in the process of step S8. The weights of the neural network are updated so that the expression (2) is minimized going back in time. As a result, the process of step S9 is completed, and the distribution rule learning process proceeds to the process of step S10.

In the process of step S10, the learning device 2 determines whether or not the event in the physical distribution environment caused by the next action of the corresponding crane is the end event. Specifically, the collision flag shown in Table 1 (a flag indicating whether or not a collision has occurred between cranes; 0: no collision; 1: collision); 0: with stagnation, 1: without stagnation), and end flag (flag indicating whether transportation of N cargoes has been completed, 0: transportation not completed, 1: transportation completed) If this value is 1, the learning device 2 determines that the occurrence event is the end event (step S10: Yes), and returns the distribution rule learning process to the process of step S2. On the other hand, when the values of the collision flag, the retention flag, and the end flag shown in Table 1 are all 0, the learning device 2 determines that the occurrence event is not the end event (step S10: No), and the learning device 2 returns the distribution rule learning process to the process of step S6.

[Model execution processing]
Next, referring to FIG. 5, the model execution processing of step S7 will be described in detail.

FIG. 5 is a flowchart showing the flow of model execution processing shown in FIG. The flowchart shown in FIG. 5 starts when the process of step S6 shown in FIG. 2 is completed, and the model execution process proceeds to the process of step S21.

In the process of step S21, the simulator unit 21 initializes the values of all flags (collision flag, stay flag, end flag, loading flag, unloading flag shown in Table 1) used in the model execution process to zero. Thereby, the process of step S21 is completed, and the model execution process proceeds to the process of step S22.

In the process of step S22, the simulator section 21 causes the corresponding crane to start traveling toward the destination designated by the next action according to the next action of the corresponding crane determined by the next action determining section 22 when learning. Thereby, the process of step S22 is completed, and the model execution process proceeds to the process of step S23.

In the processing of step S23, the simulator section 21 executes the simulation model 6 until a predetermined event occurs. Thereby, the process of step S23 is completed, and the model execution process proceeds to the process of step S24.

In the process of step S24, the simulator unit 21 determines whether or not a collision between cranes has occurred. As a result of determination, if a collision between cranes occurs (step S24: Yes), the simulator unit 21 advances the model execution process to the process of step S25. On the other hand, if the collision between the cranes has not occurred (step S24: No), the simulator unit 21 advances the model execution process to the process of step S26.

In the process of step S25, the simulator unit 21 sets the value of the collision flag to 1 (occurrence of collision). As a result, the process of step S25 is completed, and the model execution process proceeds to the process of step S39.

In the process of step S26, the simulator unit 21 determines whether or not the crane has arrived at the destination. As a result of the determination, if the crane has arrived at the destination (step S26: Yes), the simulator section 21 advances the model execution process to the process of step S30. On the other hand, if the crane has not arrived at the destination (step S26: No), the simulator unit 21 advances the model execution process to step S27.

In the process of step S27, the simulator unit 21 determines whether cargo has occurred at the generation locations P1 to P3. As a result of the determination, if cargo has occurred at the generation locations P1 to P3 (step S27: Yes), the simulator section 21 advances the model execution processing to the processing of step S28. On the other hand, when cargo has not occurred at the generation locations P1 to P3 (step S27: No), the simulator section 21 returns the model execution processing to the processing of step S23.

In the process of step S28, the simulator unit 21 determines whether or not two or more cargoes are staying at the generation locations P1 to P3. As a result of determination, when two or more cargoes are stagnating (step S28: Yes), the simulator unit 21 advances the model execution process to step S29. On the other hand, if two or more cargoes are not stagnant (step S28: No), the simulator section 21 advances the model execution process to the process of step S39.

In the process of step S29, the simulator unit 21 sets the value of the stay flag to 1 (with stay). Thereby, the processing of step S29 is completed, and the model execution processing proceeds to the processing of step S39.

In the process of step S30, the simulator unit 21 determines whether or not the crane has cargo. As a result of determination, if there is a load on the crane (step S30: Yes), the simulator unit 21 advances the model execution process to the process of step S34. On the other hand, if there is no load on the crane (step S30: No), the simulator unit 21 advances the model execution process to step S31.

In the process of step S31, the simulator unit 21 determines whether or not the arrival place of the crane is a place where cargo is present. As a result of the determination, if the arrival place of the crane is the generation place where there is cargo (step S31: Yes), the simulator unit 21 advances the model execution process to the process of step S32. On the other hand, if the destination of the crane is not the place where there is a load (step S31: No), the simulator unit 21 advances the model execution process to step S39.

In the process of step S32, the simulator unit 21 loads the cargo at the location of occurrence to the corresponding crane. As a result, the process of step S32 is completed, and the model execution process proceeds to the process of step S33.

In the processing of step S33, the simulator unit 21 sets the value of the loading flag (flag indicating whether or not the loading was successful) to 1 (successful loading). Thereby, the process of step S33 is completed, and the model execution process proceeds to the process of step S39.

In the processing of step S34, the simulator unit 21 determines whether or not the destination of the crane is the cargo destination. As a result of determination, if the destination of the crane is the cargo destination (step S34: Yes), the simulator unit 21 advances the model execution process to the process of step S35. On the other hand, if the destination of the crane is not the cargo destination (step S34: No), the simulator unit 21 advances the model execution process to step S39.

In the processing of step S35, the simulator unit 21 executes the work of loading and unloading the cargo to the destination of the corresponding crane. As a result, the process of step S35 is completed, and the model execution process proceeds to the process of step S36.

In the process of step S36, the simulator unit 21 sets the value of the unloading flag (flag indicating whether or not the cargo has been successfully unloaded) to 1 (successful unloading). Thereby, the process of step S36 is completed, and the model execution process proceeds to the process of step S37.

In the process of step S37, the simulator unit 21 determines whether transportation of N loads has been completed. As a result of the determination, if transportation of N loads has been completed (step S37: Yes), the simulator unit 21 advances the model execution process to the process of step S38. On the other hand, if transportation of N loads has not been completed (step S37: No), the simulator unit 21 advances the model execution process to the process of step S39.

In the process of step S38, the simulator unit 21 sets the value of the end flag to 1. As a result, the process of step S38 is completed, and the model execution process proceeds to the process of step S39.

In the processing of step S39, the simulator unit 21 creates state tables shown in Tables 1 to 3 based on the processing results of steps S21 to S38. As a result, the processing of step S39 is completed, and the series of model execution processing ends.

As is clear from the above description, in the physical distribution control system 1 according to one embodiment of the present invention, the learning device 2 includes the simulator section 21 that reproduces the physical distribution environment including the movement of the transportation equipment, and the transportation system in the physical distribution environment. A learning time next action determining unit 22 that decides the next action of the equipment and instructs the transportation equipment, and a simulator unit 21 that reproduces the next action instructed to the transportation equipment to interpret the state of the physical distribution environment obtained. A learning state interpretation unit 23, a reward calculation unit 24 for calculating a reward value for evaluating the next action of the transportation equipment based on the state of the logistics environment interpreted by the learning state interpretation unit 23, and a learning next action determination unit. a learning unit 25 that corrects and learns the logistics rules by inputting the next action of the transportation equipment instructed by 22, the state of the logistics environment interpreted by the learning state interpretation unit 23, and the reward value calculated by the reward calculation unit 24; , it is possible to easily create a distribution rule for a plurality of transportation equipments that involve interference between the transportation equipments. The created learning rule is stored in the learning result storage unit 31 and read by the execution device 4 side by the learning result reading unit 41 in the execution device 4 . At the time of execution, the status is grasped by the runtime status interpreting unit 42 from the physical distribution environment in the real space, and the actual behavior is determined by the runtime next action determining unit 43, and an execution command is sent to the transportation equipment or the like. As a result, physical distribution control for a plurality of transport equipment can be made possible according to the physical distribution rule learned by the learning device 2 .

Finally, an example using the present invention is presented. FIG. 6 shows an example in which the simulation model shown in FIG. 3 is used to learn a physical distribution rule for transporting 10 loads according to the present invention. In this example, the number of times of learning is 200,000, and the average reward value after learning is a positive value. This indicates that the operation according to the distribution rule created by learning has succeeded, and the cargo is being transported to the specified location while avoiding interference between multiple cranes. It proves the effect of

Although the embodiments to which the invention made by the present inventors is applied have been described above, the present invention is not limited by the descriptions and drawings forming a part of the disclosure of the present invention according to the present embodiments. That is, other embodiments, examples, operation techniques, etc. made by those skilled in the art based on this embodiment are all included in the scope of the present invention.

1 Logistics Control System 2 Learning Device 3 Common Device 4 Execution Device 5 Real Space/Second Simulator Section 6 Simulation Model 21 Simulator Section 22 Next Action Determining Section at Learning 23 State Interpreting Section at Learning 24 Reward Calculating Section 25 Learning Section 31 Learning Result Storage unit 41 Learning result reading unit 42 Execution state interpretation unit 43 Execution next action determination unit

Claims (11)

A physical distribution rule learning device for creating physical distribution rules for a plurality of transportation equipment for transporting objects to be transported,
a simulator unit that reproduces the logistics environment including the movement of the transportation equipment;
a learning-time-next-action determining unit that determines a next action of the transportation equipment in the logistics environment and instructs the transportation equipment;
a learning-time state interpretation unit that interprets the state of the logistics environment obtained by reproducing the next action instructed to the transportation equipment by the simulator unit;
a reward calculation unit that calculates a reward value for evaluating the next action of the transportation equipment based on the state of the logistics environment interpreted by the state interpretation unit during learning;
The next action of the transportation equipment instructed by the next action determination unit during learning, the state of the physical distribution environment interpreted by the state interpretation unit during learning, and the reward value calculated by the reward calculation unit are input into the physical distribution rule. a learning unit that corrects and learns
with
The physical distribution rule learning device, wherein the learning unit uses the same neural network to learn a plurality of transportation equipment whose motions are symmetrical with each other. 2. A physical distribution rule learning apparatus according to claim 1, wherein said simulator unit is a discrete event simulator that reproduces a physical distribution environment including movement of said transportation equipment. 3. The physical distribution rule learning device according to claim 2, wherein the state of the physical distribution environment interpreted by the state interpreting unit at the time of learning is the state of the physical distribution environment when a discrete event occurs. The state of the physical distribution environment at the time of the occurrence of discrete events includes the location and destination of transportation equipment, including its stopped state, the presence or absence of objects loaded on transportation equipment and their destinations, and the location of objects to be loaded on transportation equipment. 4. The physical distribution rule learning device according to claim 3, wherein a destination is included. The reward calculation unit calculates at least one of a value for correctly transporting the transport object by the transport equipment, a value for transport time within one unit of learning, and a value for transport failure within one unit of learning. 5. The physical distribution rule learning device according to any one of claims 1 to 4, wherein said reward value is calculated based on . The physical distribution rule learning device according to any one of claims 1 to 5, wherein the learning unit performs learning using a neural network. A learning result storage unit for storing learning results of the learning unit provided in the physical distribution rule learning device according to claim 1;
a learning result reading unit that reads the learning result from the learning result storage unit;
a run-time state interpretation unit that interprets the state of the physical distribution environment in real space or the state of the physical distribution environment reproduced by the second simulator;
Using the learning result read from the learning result reading unit and the state of the physical distribution environment interpreted by the runtime state interpreting unit, the next action of the transportation device in the physical distribution environment is determined and instructed to the transportation device. a runtime next action determination unit;
A physical distribution rule execution device comprising: 8. The physical distribution rule execution device according to claim 7 , wherein the state of the physical distribution environment interpreted by the runtime state interpretation unit is the state of the physical distribution environment when a discrete event occurs. The state of the physical distribution environment at the time of the occurrence of discrete events includes the location and destination of transportation equipment, including its stopped state, the presence or absence of objects loaded on transportation equipment and their destinations, and the location of objects to be loaded on transportation equipment. 9. The physical distribution rule execution device according to claim 8 , wherein a destination is included. A logistics rule learning method for creating a logistics rule for a plurality of transportation equipment for transporting objects to be transported, comprising:
a first step of reproducing the logistics environment including the movement of the transportation equipment;
a second step of determining and directing a next action of the vehicle within the logistics environment;
a third step of interpreting the state of the logistics environment obtained by reproducing the next action instructed to the transportation equipment in the first step;
a fourth step of calculating a reward value that evaluates the next action of the vehicle based on the conditions of the logistics environment interpreted in the third step;
The next action of the transportation equipment instructed in the second step, the state of the physical distribution environment interpreted in the third step, and the remuneration value calculated in the fourth step are used as inputs to correct and learn the physical distribution rule. five steps and
including
In the fifth step, a physical distribution rule learning method, wherein learning is performed using the same neural network for a plurality of transportation equipment whose movements are symmetrical with each other. A distribution rule learning program for causing a computer to execute processing for creating distribution rules for a plurality of transportation equipment for transporting objects to be transported,
a first process that reproduces a logistics environment including the movement of the transportation equipment;
a second process of determining a next action of the transportation equipment in the logistics environment and instructing the transportation equipment;
a third process of interpreting the state of the physical distribution environment obtained by reproducing the next action instructed to the transportation equipment in the first process;
a fourth process of calculating a reward value for evaluating the next action of the transportation equipment based on the state of the logistics environment interpreted in the third process;
modifying and learning the physical distribution rule using the next action of the transportation equipment instructed in the second process, the state of the physical distribution environment interpreted in the third process, and the remuneration value calculated in the fourth process as inputs; five treatments;
on the computer , and
A physical distribution rule learning program characterized in that, in the fifth processing, learning is performed using the same neural network for a plurality of transport equipment whose movements are line symmetrical with each other.

Images