KR102636029B1 - Method and system for recommending stacking position of container based on reinforcement learning - Google Patents

Abstract

The present disclosure relates to a method for recommending a container device location based on reinforcement learning, which is performed by at least one processor. The container device location recommendation method includes receiving target container data and status data of the actual container yard, receiving status data of the actual reach stacker located in the actual container yard, and using a reinforcement learning model to It includes generating container task data from the state data of the container yard and the state data of the actual reach stacker, and the reinforcement learning model is learned based on the virtual container data, the state data of the virtual container yard, and the state data of the virtual reach stacker. do.

Description

Reinforcement learning-based container device location recommendation method and system {METHOD AND SYSTEM FOR RECOMMENDING STACKING POSITION OF CONTAINER BASED ON REINFORCEMENT LEARNING}

This disclosure relates to a method and system for recommending a container device location based on reinforcement learning. Specifically, a method of recommending the device location of a target container by generating container task data using a reinforcement learning model learned from virtual data. and systems.

In the port logistics process, containers from container ships are unloaded and then placed in an inland container yard before being transported to the next destination. Container devices have significant additional costs and space limitations, so container yard space is used efficiently by stacking containers in multiple stages.

However, in the case of multi-level loading without considering the location of the container, there is a problem that transporting the container loaded at the bottom involves unnecessary rehandling work in which all cargo loaded at the top is moved to another storage area.

The present disclosure provides a reinforcement learning-based container device location recommendation method, a computer program stored in a recording medium, and a system (device) to solve the above problems.

The present disclosure may be implemented in various ways, including as a method, system (device), or computer program stored in a readable storage medium.

According to an embodiment of the present disclosure, a reinforcement learning-based container device location recommendation method performed by at least one processor includes receiving target container data and status data of an actual container yard, the actual container yard A step of receiving the state data of the actual reach stacker located in and a step of generating container work data from the target container data, the state data of the actual container yard, and the state data of the actual reach stacker using a reinforcement learning model. Including, the reinforcement learning model is learned based on virtual container data, state data of the virtual container yard, and state data of the virtual reach stacker.

According to one embodiment of the present disclosure, the actual reach stacker includes a plurality of reach stackers located in an actual container yard, and the method further includes transmitting container job data to at least one of the plurality of reach stackers.

According to one embodiment of the present disclosure, container task data includes a task type and a device location of a target container.

According to an embodiment of the present disclosure, the operation type includes the stocking of the target container, and the device location of the target container includes a location where the target container will be installed in the actual container yard.

According to one embodiment of the present disclosure, the task type includes container rehandling, and the device location of the target container includes a location at which the target container will be rehandled.

According to an embodiment of the present disclosure, the reinforcement learning model receives virtual container data, state data of the virtual container yard, and state data of the virtual reach stacker, determines virtual container operation data based on the received data, and It is learned through a learning method that transmits virtual container task data to the reach stacker and obtains rewards for the task performance results generated by the virtual reach stacker.

According to one embodiment of the present disclosure, the reinforcement learning model determines whether virtual container task data violates a preset prohibition condition.

According to an embodiment of the present disclosure, the virtual container data is randomly set, and if the virtual container task data violates the prohibition condition, the virtual container data is randomly reset.

According to one embodiment of the present disclosure, the reinforcement learning model is updated so that the reward obtained by repeating the learning method is maximized.

According to an embodiment of the present disclosure, virtual container work data is generated under conditions that limit the device rate of the virtual container yard to an arbitrary value.

According to an embodiment of the present disclosure, a method of generating a reinforcement learning-based container device location recommendation model, performed by at least one processor, includes (a) virtual container data, state data of a virtual container yard, and state data of a virtual reach stacker; setting step, (b) generating virtual container work data from the set data using a reinforcement learning model, (c) sending virtual container work data to the virtual reach stacker to generate work performance results, and (d) ) It includes the step of obtaining compensation for the results of task performance.

According to one embodiment of the present disclosure, virtual container task data includes a task type and a device location of the virtual container.

According to an embodiment of the present disclosure, the operation type includes stocking of the virtual container, and the device location of the virtual container includes a location where the virtual container is to be installed in the virtual container yard.

According to one embodiment of the present disclosure, the task type includes virtual container rehandling, and the device location of the virtual container includes the location at which the virtual container will be rehandled.

According to an embodiment of the present disclosure, the method further includes checking whether the virtual container task data violates a preset prohibition condition.

According to an embodiment of the present disclosure, the virtual container data is randomly set, and if the virtual container task data violates the prohibition condition, the virtual container data is randomly reset.

According to an embodiment of the present disclosure, the method further includes updating the reinforcement learning model so that the reward obtained by repeating steps (a), (b), (c), and (d) is maximized.

According to an embodiment of the present disclosure, virtual container work data is generated under conditions that limit the device rate of the virtual container yard to an arbitrary value.

In order to execute the above-described method according to an embodiment of the present disclosure on a computer, a computer program stored in a computer-readable recording medium is provided.

An information processing system according to an embodiment of the present disclosure includes a communication module, a memory, and at least one processor connected to the memory and configured to execute at least one computer-readable program included in the memory, and at least one program Receives the target container data and the status data of the actual container yard, receives the status data of the actual reach stacker located in the actual container yard, and uses a reinforcement learning model to collect the target container data, the status data of the actual container yard, and the actual It includes instructions for generating container task data from the state data of the reach stacker, and the reinforcement learning model is learned based on the virtual container data, the state data of the virtual container yard, and the state data of the virtual reach stacker.

According to an embodiment of the present disclosure, containers can be efficiently installed at a container yard by providing an optimal location for installing containers using a reinforcement learning model learned from virtual data.

According to an embodiment of the present disclosure, unnecessary container rehandling work can be minimized by providing an optimal location for installing containers in a container yard using a reinforcement learning model.

According to an embodiment of the present disclosure, by generating rehandling work data considering the state of the container yard as well as container receiving work data, as many containers as possible can be optimally installed in a limited space.

The effects of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned can be explained by those skilled in the art in the technical field to which this disclosure pertains from the description of the claims (hereinafter referred to as 'the person skilled in the art'). can be clearly understood.

Embodiments of the present disclosure will be described with reference to the accompanying drawings described below, in which like reference numerals indicate like elements, but are not limited thereto.
1 is a diagram illustrating an example of using a reinforcement learning-based container device location recommendation system according to an embodiment of the present disclosure.
Figure 2 is a schematic diagram showing a configuration in which an information processing system according to an embodiment of the present disclosure is connected to communicate with a plurality of reach stackers.
Figure 3 is a block diagram showing the internal configuration of an information processing system according to an embodiment of the present disclosure.
Figure 4 is a block diagram showing the internal configuration of a processor of an information processing system according to an embodiment of the present disclosure.
Figure 5 is a diagram showing the internal configuration of a learning unit according to an embodiment of the present disclosure.
Figure 6 is a diagram showing an example of a reinforcement learning model according to an embodiment of the present disclosure.
Figure 7 is a diagram showing an example of an agent structure according to an embodiment of the present disclosure.
Figure 8 is a diagram showing an example of a method for learning a reinforcement learning model according to an embodiment of the present disclosure.
Figure 9 is a diagram illustrating an example of generating container warehousing work data according to an embodiment of the present disclosure.
FIG. 10 is a diagram illustrating an example of generating container rehandling job data according to an embodiment of the present disclosure.
Figure 11 is a flowchart showing a reinforcement learning-based container device location recommendation method according to an embodiment of the present disclosure.
Figure 12 is a flowchart showing a method for generating a reinforcement learning-based container device location recommendation model according to an embodiment of the present disclosure.

Hereinafter, specific details for implementing the present disclosure will be described in detail with reference to the attached drawings. However, in the following description, detailed descriptions of well-known functions or configurations will be omitted if there is a risk of unnecessarily obscuring the gist of the present disclosure.

In the accompanying drawings, identical or corresponding components are given the same reference numerals. Additionally, in the description of the following embodiments, overlapping descriptions of identical or corresponding components may be omitted. However, even if descriptions of components are omitted, it is not intended that such components are not included in any embodiment.

Advantages and features of the disclosed embodiments and methods for achieving them will become clear by referring to the embodiments described below in conjunction with the accompanying drawings. However, the present disclosure is not limited to the embodiments disclosed below and may be implemented in various different forms. The present embodiments are merely intended to ensure that the present disclosure is complete and that the present disclosure does not convey the scope of the invention to those skilled in the art. It is provided only for complete information.

Terms used in this specification will be briefly described, and the disclosed embodiments will be described in detail. The terms used in this specification are general terms that are currently widely used as much as possible while considering the function in the present disclosure, but this may vary depending on the intention or precedent of a technician working in the related field, the emergence of new technology, etc. In addition, in certain cases, there are terms arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the description of the relevant invention. Accordingly, the terms used in this disclosure should be defined based on the meaning of the term and the overall content of the present disclosure, rather than simply the name of the term.

In this specification, singular expressions include plural expressions, unless the context clearly specifies the singular. Additionally, plural expressions include singular expressions, unless the context clearly specifies plural expressions. When it is said that a certain part includes a certain element throughout the specification, this does not mean excluding other elements, but may further include other elements, unless specifically stated to the contrary.

Additionally, the term 'module' or 'unit' used in the specification refers to a software or hardware component, and the 'module' or 'unit' performs certain roles. However, 'module' or 'unit' is not limited to software or hardware. A 'module' or 'unit' may be configured to reside on an addressable storage medium and may be configured to run on one or more processors. Thus, as an example, a 'module' or 'part' refers to components such as software components, object-oriented software components, class components and task components, processes, functions and properties. , procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, or variables. Components and 'modules' or 'parts' may be combined into smaller components and 'modules' or 'parts' or further components and 'modules' or 'parts'. Could be further separated.

According to an embodiment of the present disclosure, a 'module' or 'unit' may be implemented with a processor and memory. 'Processor' should be interpreted broadly to include general-purpose processors, central processing units (CPUs), microprocessors, digital signal processors (DSPs), controllers, microcontrollers, state machines, etc. In some contexts, 'processor' may refer to an application-specific integrated circuit (ASIC), programmable logic device (PLD), field programmable gate array (FPGA), etc. 'Processor' refers to a combination of processing devices, for example, a combination of a DSP and a microprocessor, a combination of a plurality of microprocessors, a combination of one or more microprocessors in combination with a DSP core, or any other such combination of configurations. You may. Additionally, 'memory' should be interpreted broadly to include any electronic component capable of storing electronic information. 'Memory' refers to random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), programmable read-only memory (PROM), erasable-programmable read-only memory (EPROM), May also refer to various types of processor-readable media, such as electrically erasable PROM (EEPROM), flash memory, magnetic or optical data storage, registers, etc. A memory is said to be in electronic communication with a processor if the processor can read information from and/or write information to the memory. The memory integrated into the processor is in electronic communication with the processor.

In this disclosure, 'Reinforcement Learning (RL)' is an area of machine learning in which a computer selects an action that maximizes the reward among selectable actions for a given state. It can refer to a learning method. Here, the computer program that is the target of reinforcement learning can be referred to as an agent, and the agent can explore a given environment and establish a policy that indicates the probability of an action it can take in the current state. there is. Additionally, when a specific state is given to the agent by a given environment (the agent can be seen as obtaining the state through observation), the agent takes action according to the state, and the environment gives the agent a reward. do. Through this process, the agent can interact with the environment and learn actions or policies that can result in high rewards. In other words, the agent or its policy is learned by interacting states, actions, and rewards between the given environment and the agent, and the goal of reinforcement learning may be to establish a policy through which the agent can receive the maximum reward.

In this disclosure, 'machine learning model' may include any model used to infer an answer to a given input. According to one embodiment, the machine learning model may include an artificial neural network model including an input layer (layer), a plurality of hidden layers, and an output layer. Here, each layer may include multiple nodes. In this disclosure, a machine learning model may refer to an artificial neural network model, and an artificial neural network model may refer to a machine learning model.

In this disclosure, 'container yard' may refer to a place where containers are stored, delivered, and received. For example, it may refer to a yard located in an area near a port where a container ship shipping company accumulates, stores, and installs containers, and transfers loaded containers. Additionally, 'container yard' may be referred to as a container storage yard.

In the present disclosure, 'reach stacker' may refer to equipment mainly used to move empty containers within a dock or container storage yard that handles small containers, or to unload container boxes from a container crane and install them within the container storage facility. there is.

Figure 1 is a diagram illustrating an example in which the reinforcement learning-based container device location recommendation system 110 according to an embodiment of the present disclosure is used. The container device location recommendation system 110 may be configured to include a reinforcement learning model 120.

According to one embodiment, the container device location recommendation system 110 may receive target container data 132. Target container data 132 may refer to data associated with a container to be installed in an actual container yard or a container to be rehandled. For example, the target container data 132 may include the port of entry of the container ship that loaded the target container, the port arrival date, the operation route information of the container ship, the shipping company information of the container ship, the size of the container, the shipping date of the container, etc. It may include etc.

According to one embodiment, the container device location recommendation system 110 may receive status data 134 of the actual container yard. The status data 134 of the actual container yard may refer to the current status of the actual container yard and data about containers being installed in the actual container yard. For example, the status data 134 of the actual container yard includes the size of the container yard, the device amount at the maximum device rate of the container yard, whether a container is being installed in the container yard, and if a container is being installed in the container yard, the status of the container. It may include detailed information, etc.

According to one embodiment, the container device location recommendation system 110 may receive status data 136 of the actual reach stacker. In this case, the status data 136 of the actual reach stacker may refer to data related to the current status of a plurality of reach stackers located in the actual container yard. For example, the status data 136 of the actual reach stacker may include the location of the actual reach stacker, whether the actual reach stacker is working, the amount of actual reach stacker work, the driver information of the actual reach stacker, and other information about the actual reach stacker. You can.

According to one embodiment, the container device location recommendation system 110 uses the reinforcement learning model 120 to collect target container data 132, state data 134 of the actual container yard, and state data 136 of the actual reach stacker. Container task data 140 can be generated from. In this case, the container task data 140 may include the task type and device location of the target container. For example, the container operation data 140 may include container warehousing operation data including container warehousing as a task type and a location where the target container will be installed in the actual container yard as a container device location. As another example, the container task data 140 may include container rehandling task data including container rehandling as a task type and a location at which the target container is to be rehandled as a container device location. Details about this will be described later with reference to FIGS. 9 and 10.

According to one embodiment, the container device location recommendation system 110 may transmit the generated container task data 140 to the reach stacker 150 and instruct it to perform the corresponding container task. In this case, the reach stacker 150 may be one of a plurality of reach stackers existing in an actual container yard. In FIG. 1 , the container work data 140 is shown as being transmitted to one reach stacker 150, but the present invention is not limited thereto, and the container work data 140 may be transmitted to a different number of reach stackers.

According to one embodiment, the reinforcement learning model 120 may be learned based on virtual container data, state data of a virtual container yard, and state data of a virtual reach stacker. In this case, the reinforcement learning model 120 may include an arbitrary algorithm learned to select the container device location that gives the highest reward. For example, the reinforcement learning model 120 may be learned using a machine learning model (eg, artificial neural network model, etc.). Details about this will be described later with reference to FIGS. 6 to 8.

Through the configuration described above, the container device location recommendation system 110 according to some embodiments of the present disclosure provides the optimal location for installing containers in the container yard using a reinforcement learning model, thereby efficiently storing containers in the container yard. You can minimize unnecessary container rehandling work.

Figure 2 is a schematic diagram showing a configuration in which the information processing system 230 according to an embodiment of the present disclosure is connected to communicate with a plurality of reach stackers 212_1, 212_2, and 212_3. In Figure 2, the target object is shown as a reach stacker (212_1, 212_2, 212_3), but the target object is not limited to this, and the target object may be a driver operating the vehicle or a user terminal used by the driver. Additionally, the information processing system 230 may include the container device location recommendation system(s) described with reference to FIG. 1 .

In one embodiment, information processing system 230 may include one or more server devices and/or capable of storing, providing, and executing computer-executable programs (e.g., downloadable applications) and data related to recommending container device locations. It may include a database, or one or more distributed computing devices and/or distributed databases based on cloud computing services. The information processing system 230 may provide information corresponding to a signal input through an application (eg, an application related to container device location, etc.) or perform corresponding processing. For example, the information processing system 230 may transmit container task data to a plurality of reach stackers 212_1, 212_2, and 212_3 through any application related to the container device location.

The information processing system 230 may communicate with a plurality of reach stackers 212_1, 212_2, and 212_3 through the network 220. The network 220 may be configured to enable communication between a plurality of reach stackers 212_1, 212_2, and 212_3 and the information processing system 230. Depending on the installation environment, the network 220 may be, for example, a wired network such as Ethernet, a wired home network (Power Line Communication), a telephone line communication device, and RS-serial communication, a mobile communication network, a wireless LAN (WLAN), It may consist of wireless networks such as Wi-Fi, Bluetooth, and ZigBee, or a combination thereof. The communication method is not limited, and may include communication methods utilizing communication networks that the network 220 may include (e.g., mobile communication networks, wired Internet, wireless Internet, broadcasting networks, satellite networks, etc.), as well as reach stacker (212_1, 212_2, 212_3) ) may also include short-range wireless communication between

In FIG. 2 , three reach stackers (212_1, 212_2, 212_3) are shown as communicating with the information processing system 230 through the network 220, but this is not limiting, and a different number of reach stackers may be connected to the network 220. It may be configured to communicate with the information processing system 230 through.

In one embodiment, reach stackers 212_1, 212_2, and 212_3 transmit requests for data related to container operations to information processing system 230 via network 220 and retrieve data related to container operations from information processing system 230. can receive. In response to receiving data related to container work, the reach stackers 212_1, 212_2, and 212_3 may be operated to work on the corresponding container.

Figure 3 is a block diagram showing the internal configuration of the information processing system 230 according to an embodiment of the present disclosure. The information processing system 230 for container device location recommendation may include a memory 310, a processor 320, a communication module 330, and an input/output interface 340. As shown in FIG. 3, the information processing system 230 may be configured to communicate information and/or data over a network using a communication module 330.

Memory 310 may include any non-transitory computer-readable recording medium. According to one embodiment, the memory 310 may include a non-permanent mass storage device, such as a disk drive, solid state drive (SSD), or flash memory. As another example, non-perishable mass storage devices such as ROM, SSD, flash memory, disk drive, etc. may be included in the information processing system 230 as a separate persistent storage device that is distinct from memory. Additionally, the memory 310 may store an operating system and at least one program code (eg, a container task data generation command installed and driven in the information processing system 230, etc.). In FIG. 3, the memory 310 is shown as a single memory, but this is only for convenience of explanation, and the memory 310 may include a plurality of memories and/or buffer memories.

These software components may be loaded from a computer-readable recording medium separate from the memory 310. Recording media readable by such a separate computer may include recording media directly connectable to the information processing system 230, for example, floppy drives, disks, tapes, DVD/CD-ROM drives, memory cards, etc. It may include a recording medium that can be read by a computer. As another example, software components may be loaded into the memory 310 through the communication module 330 rather than a computer-readable recording medium. For example, at least one program is a computer program (e.g., container task data generation) installed by files provided through the communication module 330 by developers or a file distribution system that distributes the installation file of the application. It may be loaded into the memory 310 based on a program (for example, program, etc.).

The processor 320 may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input/output operations. Commands may be provided to a user terminal (not shown) or another external system by the memory 310 or the communication module 330. For example, the processor 320 may train a reinforcement learning model through virtual container yard status data, etc.

The communication module 330 may provide a configuration or function for a user terminal (not shown) and the information processing system 230 to communicate with each other through a network, and the information processing system 230 may be connected to an external system (for example, a separate Configuration or functions for communicating with cloud systems, etc. can be provided. For example, control signals, commands, data, etc. provided under the control of the processor 320 of the information processing system 230 pass through the communication module 330 and the network to the user through the communication module of the user terminal and/or external system. It may be transmitted to the terminal and/or external system. For example, an external system (eg, a reach stacker's information processing system) may receive container task data, etc. from the information processing system 230.

Additionally, the input/output interface 340 of the information processing system 230 is connected to the information processing system 230 or means for interfacing with a device (not shown) for input or output that the information processing system 230 may include. It can be. For example, the input/output interface 340 may include at least one of a PCI express interface and an Ethernet interface. In FIG. 3 , the input/output interface 340 is shown as an element configured separately from the processor 320, but the present invention is not limited thereto, and the input/output interface 340 may be included in the processor 320. Information processing system 230 may include more components than those in FIG. 3 . However, there is no need to clearly show most prior art components.

The processor 320 of the information processing system 230 may be configured to manage, process, and/or store information and/or data received from a plurality of user terminals and/or a plurality of external systems. According to one embodiment, the processor 320 may receive status data of an actual container yard, etc. In FIG. 3, the processor 320 is shown as a single processor, but this is only for convenience of explanation, and the processor 320 may include a plurality of processors.

Figure 4 is a block diagram showing the internal configuration of the processor 320 of the information processing system according to an embodiment of the present disclosure. The processor 320 may include a data receiving unit 410, a learning unit 420, and a container yard master 430. In this case, programs related to the data receiving unit 410, learning unit 420, and container yard master 430 are stored or loaded in any storage medium (e.g., memory 310, etc.), and are processed by the processor 320. Can be accessed or executed. In FIG. 4 , each component of the processor 320 represents functionally distinct functional elements, and a plurality of components may be implemented in a form that is integrated with each other in an actual physical environment. In addition, in Figure 4, the processor 320 is implemented by dividing into a data receiving unit 410, a learning unit 420, and a container yard master 430, but it is not limited to this, and some components may be omitted or other components may be added. .

According to one embodiment, the data receiving unit 410 may receive data for generating container work data. For example, the data receiver 410 may receive target container data (e.g., port of entry of the container ship loading the target container, port arrival date, etc.), status data of the actual container yard (e.g., size of the container yard, maximum device of the container yard, etc.) It is possible to receive data on the status of the actual reach stacker (e.g., location of the reach stacker, whether the reach stacker is working, etc.). Additionally, the data receiving unit 410 may transmit the received data to the container yard master 430.

According to one embodiment, the learning unit 420 may learn a reinforcement learning model of the container yard master 430 based on virtual container data, state data of the virtual container yard, and state data of the virtual reach stacker. In more detail, the learning unit 420 generates virtual container work data using a reinforcement learning model and updates the reinforcement learning model to maximize the reward obtained based on this, thereby creating the reinforcement learning model of the container yard master 430. It can be learned. Details of an example of the internal configuration of the learning unit 420 will be described later with reference to FIG. 5 . Additionally, examples of the learning method of the learning unit 420 will be described later with reference to FIGS. 6 to 8.

According to one embodiment, the container yard master 430 may generate container operation data from target container data, status data of the actual container yard, and status data of the actual reach stacker. To this end, the container yard master 430 may be configured to include a reinforcement learning model. Additionally, the container yard master 430 may perform container work by transmitting container work data generated for at least one of the plurality of reach stackers located in the actual container yard through the communication module of the information processing system.

FIG. 5 is a diagram showing the internal configuration of the learning unit 420 according to an embodiment of the present disclosure. The learning unit 420 may include a virtual container setting unit 510, a virtual container yard setting unit 520, a virtual reach stacker control unit 530, and a virtual container yard master 540. In this case, programs related to the virtual container setting unit 510, the virtual container yard setting unit 520, the virtual reach stacker control unit 530, and the virtual container yard master 540 are stored in any storage medium (e.g., memory 310). etc.) and may be accessed or executed by the processor 320. In FIG. 5, each component of the learning unit 420 represents functional elements that are functionally distinct, and a plurality of components may be implemented in a form that is integrated with each other in an actual physical environment. In addition, in Figure 5, the learning unit 420 is implemented by dividing into a virtual container setting unit 510, a virtual container yard setting unit 520, a virtual reach stacker control unit 530, and a virtual container yard master 540, but is limited to this. Some configurations may be omitted or other configurations may be added. For example, the virtual container yard master 540 may be omitted from the learning unit 420, and the learning unit 420 directly processes the processor's container yard master and learning data (e.g., virtual container data, status of the virtual container yard). By sending and receiving data (state data of the virtual reach stacker), the processor's container yard master can be learned.

According to one embodiment, the virtual container setting unit 510 may set virtual container data for training a reinforcement learning model. Virtual container data may be set as data for a virtual container corresponding to target container data. The virtual container data may include the port of entry of the virtual container ship, the port arrival date, the operation route information of the virtual container ship, and the shipping company information of the virtual container ship. Additionally, the virtual container setting unit 510 may transmit the set virtual container data to the virtual container yard master 540.

According to one embodiment, the virtual container yard setting unit 520 may set status data of the virtual container yard. A virtual container yard may refer to a container yard in a virtual space that implements an actual container yard in a virtual environment. For example, a virtual container yard may include one or more container yard sections, and there may be roads between the container yard sections for passage by reach stackers, tractors, and container trailers. The configuration of the virtual container yard section can be changed to be the same as the environment of the actual container yard. Additionally, the status data of the virtual container yard may be data corresponding to the status data of the actual container yard. The status data of the virtual container yard includes the size of the virtual container yard, the device amount at the maximum device rate of the virtual container yard, whether a virtual container is being installed in the virtual container yard, and if a virtual container is being installed in the virtual container yard, the status of the corresponding virtual container. It may include detailed information, etc. The virtual container yard setting unit 520 may transmit status data of the set virtual container yard to the virtual container yard master 540.

According to one embodiment, the virtual reach stacker control unit 530 may set status data of the virtual reach stacker yard. A virtual reach stacker may refer to a virtual object that implements an actual reach stacker in a virtual environment. For example, a plurality of virtual reach stackers may exist in a virtual container yard, and each of the plurality of virtual reach stackers may have one attribute of a running state, a working state, or a container transporting state. The number of virtual reach stackers can be changed to be the same as the number of actual reach stackers existing in the actual container yard. Additionally, the status data of the virtual reach stacker may be data corresponding to the status data of the actual reach stacker existing in the actual container yard. The status data of the virtual reach stacker may include the location of the virtual reach stacker, whether the virtual reach stacker is working, the workload of the virtual reach stacker, driver information of the virtual reach stacker, and other information about the actual reach stacker. The virtual reach stacker control unit 530 may transmit status data of the set virtual reach stacker to the virtual container yard master 540.

According to one embodiment, the virtual reach stacker control unit 530 may receive virtual container work data generated by the virtual container yard master 540 and control the virtual reach stacker. The virtual reach stacker can perform device, unloading, and rehandling tasks according to virtual container work data.

According to one embodiment, the virtual container yard master 540 may generate virtual container work data from virtual container data, state data of the virtual container yard, and state data of the virtual reach stacker. To this end, the virtual container yard master 540 may be configured to include a reinforcement learning model. Additionally, the virtual container yard master 540 may transmit the generated virtual container work data to the virtual reach stacker control unit 530 to perform container work in a virtual environment.

According to one embodiment, the reinforcement learning model of the virtual container yard master 540 may be learned through a series of learning methods. For example, the reinforcement learning model of the virtual container yard master 540 receives virtual container data, state data of the virtual container yard, and state data of the virtual reach stacker, and generates virtual container operation data based on the received data. , It can be learned through a learning method that transmits virtual container work data to the virtual reach stacker and obtains compensation for the task performance results generated by the virtual reach stacker. Additionally, the reinforcement learning model can be updated to maximize the reward obtained by repeating this learning method. Details about this will be described later with reference to FIG. 8. The reinforcement learning model of the virtual container yard master 540 learned through this process can update the reinforcement learning model of the container yard master to generate optimal container work data.

Figure 6 is a diagram showing an example of a reinforcement learning model according to an embodiment of the present disclosure. Reinforcement Learning (RL) is a field of machine learning and refers to a learning method in which an agent 610 selects an action that maximizes reward among selectable actions for the current state in a given environment 600. You can. Here, the agent 610 may include an algorithm and/or a machine learning model (e.g., artificial neural network model, etc.) that are subject to reinforcement learning, and may explore the given environment to determine the actions it can take in the current state. A policy representing probability can be established.

In this disclosure, reinforcement learning may include any of the algorithms that can handle continuous actions. In one embodiment, the agent may be trained using policy-based reinforcement learning (policy-based RL). Here, policy-based reinforcement learning may refer to a learning method that directly models a policy as a function using a neural network. When a state is input to the policy neural network modeled in this way, an action in response to it can be directly output.

In another embodiment, the agent may be trained using an actor-critic algorithm. Here, the actor-critic algorithm can determine actions through policy-based reinforcement learning and can help learn this policy using a value function. For example, Advantage Actor-Critic (A2C), Asynchronous Advantage Actor-Critic (A3C), Proximal Policy Optimization (PPO), etc. may be included in the actor-critic algorithm.

That is, the present disclosure may include both a policy-based reinforcement learning algorithm and an actor-critic algorithm, and both methods may use policy gradient (PG). Here, policy gradient may refer to a method of gradually changing policy parameters so that the agent can receive more rewards. Through this, the modeled policy neural network can be learned as an optimal policy.

These policy gradient algorithms include Stochastic Policy Gradient Optimization, which outputs the probability of an action for a given state, and Deterministic Policy Gradient Optimization, which directly outputs the action for a given state. may include. Here, the algorithm that outputs the probability of action (probability policy gradient) may include A2C (Advantage Actor-Critic), SAC (Soft Actor Critic), PPO (Proximal Policy Optimization), TRPO (Trust Region Policy Optimization), etc. Algorithms (decision policy gradients) that directly output actions may include, but are not limited to, DDPG (Deep Deterministic Policy Gradient) and TD3 (Twin Delayed Deep Deterministic Policy Gradient).

In this disclosure, state 620 may refer to current information of a target object. Here, the current information may refer to current information of the container, current information of the container yard, or current information of the reach stacker. Action 640 may include producing container task data. In addition, the reward 630 is a benefit obtained for the action of the target object, and may refer to the weight of the agent's machine learning model (e.g., artificial neural network model) that is updated when the target object produces container work data. It is not limited, and parameters that can maximize the objective function of reinforcement learning can be updated through compensation. Environment 600 refers to any environment that represents or characterizes container information, container yard information, or reach stacker information, for example, the port of entry of the container ship loading the container, the port arrival date, and the port of entry of the container ship loading the container. It may include port of entry, date of entry, location of the reach stacker, and whether the reach stacker is in operation.

According to one embodiment, when the agent 610 receives a specific state from a given environment 600 (or the agent acquires a specific state through monitoring), the agent 610 generates a control command according to the state and , the generated control command can be transmitted to the target object of the environment 600. In response, the target object in the environment 600 may provide the agent 610 with a reward 630 for the action. By repeating this process, the agent 610 can learn to interact with the environment 600 and determine actions that can result in high rewards.

According to one embodiment, in order to express the object of the above-described reinforcement learning, MDP (Markov Decision Process), which models the decision-making process, may be used. MDP is a decision-making model that adds the concepts of reward (630), action (640), and policy to the Markov Process, which represents the process in which the state changes stochastically as time progresses.

Accordingly, the navigation task M of the target object based on the MDP can be expressed as Equation 1.

In task M of Equation 1 above, O may represent an observation, and A may represent the space of the action 640. Here, the observation value may represent any information associated with the state of the target object that may refer to the state 620, for example, current information of the container, current information of the container yard, or current information of the reach stacker, etc. Action 640 may represent a control command for container task data. Additionally, in task M, r may represent a reward function for the action of the target object. Additionally, in task M, each of P and γ can represent the state transition probability and decay rate.

Figure 7 is a diagram showing an example of the structure of an agent according to an embodiment of the present disclosure. According to one embodiment, the agent may use an artificial neural network model as a machine learning model to output behavior from environmental data. Here, in the artificial neural network model, as in a biological neural network, nodes, which are artificial neurons that form a network through the combination of synapses, repeatedly adjust the weights of the synapses, creating a gap between the correct output corresponding to a specific input and the inferred output. By learning to reduce errors, a machine learning model with problem-solving capabilities can be expressed. For example, the artificial neural network model may include random probability models, neural network models, etc. used in artificial intelligence learning methods such as machine learning and deep learning. The artificial neural network model is implemented as a multilayer perceptron (MLP), which consists of multiple layers of nodes and connections between them. The artificial neural network model according to this embodiment can be implemented using one of various artificial neural network model structures including MLP. The learning method of the artificial neural network model includes a supervised learning method that learns to optimize problem solving by inputting a teacher signal (correct answer), and an unsupervised learning method that does not require a teacher signal. there is.

As shown in Figure 7, the agent includes an input layer 710 that receives input signals or data from the outside, an output layer 730 that outputs an output signal or data corresponding to the input data, an input layer 710, and an output layer ( 730) and consists of a hidden layer 720 that receives signals from the input layer 710, extracts characteristics, and transmits them to the output layer 730. Here, the output layer 730 receives a signal from the hidden layer 720 and outputs it to the outside. In one embodiment, an LSTM 722 may be used as part of the hidden layer 720. LSTM can refer to the structure of a neural network designed to enable short- and long-term memory by compensating for the disadvantage of existing RNNs in that they cannot remember information located far from the output. It can be mainly used for time series processing or natural language processing. .

According to one embodiment, the agent's input variables may include environmental data. In this case, the environmental data may include virtual container data, virtual container yard status data, and virtual reach stacker status data. In this way, when the above-mentioned input variables are input through the input layer 710, the output variables output from the agent's output layer 730 may include container work data corresponding to actions and work instructions.

In this way, a plurality of input variables and a plurality of output variables corresponding to the agent's input layer 710 and output layer 730 are matched, respectively, and the input variables included in the input layer 710, the hidden layer 720, and the output layer 730 are By adjusting the synapse values between nodes, they can be learned so that the correct output corresponding to a specific input can be extracted. Through this learning process, the hidden characteristics of the agent's input variables can be identified and the synapse values (or weights) between the agent's nodes can be adjusted to reduce the error between the output variables calculated based on the input variables and the target output. You can. In one embodiment, the computing device receives environmental data and trains the agent to minimize the loss of the first virtual container work data, which is the ground truth, and the second virtual container work data output from the agent. You can. Using this learned agent, virtual container task data can be automatically generated from virtual container data, virtual container yard state data, and virtual reach stacker state data, and the generated virtual container task data can be used for reinforcement learning. You can.

FIG. 8 is a diagram illustrating an example of a method 800 for learning a reinforcement learning model according to an embodiment of the present disclosure. Method 800 may be performed by at least one processor (e.g., processor 320) of an information processing system. Additionally, the method 800 can be used to learn a reinforcement learning model of the virtual container master or a reinforcement learning model of the container master. As described later, the reinforcement learning model is a reinforcement learning model of the virtual container master or a reinforcement learning model of the container master. can refer to.

As shown, the method 800 may be initiated by randomly setting virtual container data (S810). Additionally, status data of the virtual container yard and status data of the virtual reach stacker may be set. In this case, when setting the status data of the virtual container yard, the device rate of the virtual container yard may be limited to an arbitrary value. For example, the status data of a virtual container yard may limit the receipt of virtual containers to only 70% of the maximum number of devices. The reinforcement learning model can receive set virtual container data, state data of the virtual container yard, and state data of the virtual reach stacker.

Afterwards, it can be determined whether the reinforcement learning model is initially trained (S820). If learning of a reinforcement learning model is started for the first time without a previous model, an initial model with the weight value of the layer set randomly can be created (S832). On the other hand, if it is not initial learning, learning can proceed using the model with the existing weights set (S834).

Afterwards, virtual container task data may be generated from the reinforcement learning model (S840). Virtual container task data may include the task type and device location of the virtual container. For example, the task type includes the receipt of a virtual container, and the device location of the virtual container includes a location where the virtual container will be installed in the virtual container yard. Virtual container receipt operation data may be generated. As another example, virtual container rehandling data may be generated where the task type includes container rehandling and the virtual container device location includes a location at which the virtual container will be rehandled. Virtual container receiving operation data and virtual container rehandling data can be generated together or selectively. Additionally, virtual container work data may be generated under conditions that limit the device rate of the virtual container yard to an arbitrary value. For example, virtual container operation data can be generated under the condition that containers are received only in quantities equivalent to 70% of the maximum number of devices in the virtual container yard.

Afterwards, it can be checked whether the virtual container task data violates a preset prohibition condition. More specifically, it may be determined whether the virtual container is prohibited in the environment (S850). For example, if a container located at the bottom of the device location of a virtual container is exported earlier than the corresponding virtual container, a rehandling operation will be definitely performed in the future. Whether the container located at the bottom will be exported earlier than the corresponding virtual container. can be judged. If the virtual container task data violates a preset prohibition condition, the reinforcement learning model may be updated to learn that the corresponding virtual container task data is a prohibited behavior (S852). Additionally, virtual container data may be randomly reset.

If the virtual container task data does not violate preset prohibition conditions, the generated virtual container task data can be transmitted to the virtual reach stacker. The virtual reach stacker can generate task performance results from virtual container task data, and the reinforcement learning model can obtain rewards for the task performance results (S860). Finally, the reinforcement learning model can be updated so that the reward obtained by repeatedly performing steps S810 to S860 described above is maximized (S870).

Figure 9 is a diagram illustrating an example of generating container warehousing work data 930 according to an embodiment of the present disclosure. The container yard master 430 may generate container receiving operation data 930 by receiving target container data 922, status data 924 of the actual container yard, and status data 926 of the actual reach stacker. For this purpose, the container yard master 430 may include a reinforcement learning model 120. The target container data 922, the status data 924 of the actual container yard, and the status data 926 of the actual reach stacker are the target container data 132, the status data 134 of the actual container yard, and the actual reach stacker disclosed herein. It may correspond to the status data 136 of the stacker. In FIG. 9, the data received by the container yard master 430 is shown divided into target container data 922, status data of the actual container yard 924, and status data of the actual reach stacker 926, but is not limited to this. Some data may be omitted or other data may be added. Additionally, a plurality of data may be integrated with each other and received as one data.

According to one embodiment, target container data 922 may refer to data associated with a container to be installed in an actual container yard. For example, the target container data 922 may include the port of entry of the container ship that loaded the target container, the port arrival date, the operation route information of the container ship, and the shipping company information of the container ship.

According to one embodiment, the status data 924 of the actual container yard may refer to the current state of the actual container yard and data about containers being installed in the actual container yard. For example, the status data 924 of the actual container yard includes the size of the container yard, the device amount at the maximum device rate of the container yard, whether a container is being installed in the container yard, and if a container is being installed in the container yard, the status of the container. It may include detailed information, etc.

According to one embodiment, the state data 926 of the actual reach stacker may refer to data associated with the current state of a plurality of reach stackers located in an actual container yard. For example, the status data 926 of the actual reach stacker may include the location of the actual reach stacker, whether the actual reach stacker is working, the amount of actual reach stacker work, the driver information of the actual reach stacker, and other information about the actual reach stacker. You can.

According to one embodiment, the container warehousing operation data 930 may include data indicating that the operation type is warehousing and a location where the target container will be installed in the actual container yard. The container yard master 430 may transmit the container receiving operation data 930 to the reach stacker 940 to perform the operation. The reach stacker 940 that has received the data may perform the task of moving the target container from the actual container yard to the location where the target container will be installed.

FIG. 10 is a diagram illustrating an example of generating container rehandling job data 1030 according to an embodiment of the present disclosure. The container yard master 430 may generate container rehandling job data 1030 by receiving target container data 1022, status data 1024 of the actual container yard, and status data 1026 of the actual reach stacker. For this purpose, the container yard master 430 may include a reinforcement learning model 120. The target container data 1022, the status data 1024 of the actual container yard, and the status data 1026 of the actual reach stacker are the target container data 132, the status data 134 of the actual container yard, and the actual reach stacker disclosed herein. It may correspond to the status data 136 of the stacker. In FIG. 10, the data received by the container yard master 430 is shown divided into target container data 1022, actual container yard status data 1024, and actual reach stacker status data 1026, but is not limited thereto. Some data may be omitted or other data may be added. For example, the container yard master 430 may generate container rehandling job data 1030 by searching for target containers from the status data 1024 of the actual container yard and the status data 1026 of the actual reach stacker. Additionally, a plurality of data may be integrated with each other and received as one data.

According to one embodiment, target container data 1022 may refer to data associated with a container to be rehandled in an actual container yard. For example, the target container data 1022 may include the size of the container to be rehandled, the shipping date of the container to be rehandled, etc.

According to one embodiment, the status data 1024 of the actual container yard may refer to the current state of the actual container yard and data about containers being installed in the actual container yard. For example, the status data 1024 of the actual container yard includes the size of the container yard, the device amount at the maximum device rate of the container yard, whether a container is being installed in the container yard, and if a container is being installed in the container yard, the status of the container. It may include detailed information, etc.

According to one embodiment, the state data 1026 of the actual reach stacker may refer to data associated with the current state of a plurality of reach stackers located in an actual container yard. For example, the status data 1026 of the actual reach stacker may include the location of the actual reach stacker, whether the actual reach stacker is working, the amount of actual reach stacker work, the driver information of the actual reach stacker, and other information about the actual reach stacker. You can.

According to one embodiment, the container rehandling operation data 1030 includes data indicating that the operation type is rehandling and the location in the actual container yard where the target container will be rehandled (e.g., the location where the target container will be temporarily installed, etc.) can do. The container yard master 430 may transmit container rehandling work data 1030 to the reach stacker 1040 that will perform the work. The reach stacker 1040 that has received the data may perform the task of moving the target container from the actual container yard to the location where the target container will be rehandled.

Figure 11 is a flowchart showing a reinforcement learning-based container device location recommendation method 1100 according to an embodiment of the present disclosure. Method 1100 may be performed by at least one processor (eg, processor 320) of an information processing system. As shown, the method 1100 may be initiated by receiving target container data and status data of the actual container yard (S1110). Then, the processor may receive status data of the actual reach stacker located in the actual container yard (S1120).

Finally, the processor can use a reinforcement learning model to generate container task data from target container data, state data of the actual container yard, and state data of the actual reach stacker (S1130). Container task data may include the task type and device location of the target container. In one embodiment, the operation type may include stocking of the target container, and the device location of the target container may include the location at which the target container will be installed in the actual container yard. In another embodiment, the task type may include container rehandling, and the device location of the target container may include the location at which the target container will be rehandled.

In one embodiment, the reinforcement learning model receives virtual container data, state data of the virtual container yard, and state data of the virtual reach stacker, generates virtual container operation data based on the received data, and sends the virtual container to the virtual reach stacker. It can be learned through a learning method that transmits task data and obtains rewards for the task performance results generated by the virtual reach stacker. In this case, the virtual container data can be set randomly. Additionally, virtual container work data may be generated under conditions that limit the device rate of the virtual container yard to an arbitrary value.

In one embodiment, the reinforcement learning model may determine whether virtual container task data violates a preset prohibition condition. If virtual container task data violates prohibition conditions, virtual container data may be reset randomly. Additionally, the reinforcement learning model can be updated to maximize the reward obtained by repeating the learning method.

In one embodiment, the reach stacker may include a plurality of reach stackers located in an actual container yard. Additionally, the processor may transmit container task data to at least one of the plurality of reach stackers.

The flowchart shown in FIG. 11 and the above description are only examples and may be implemented differently in some embodiments. For example, in some embodiments, the order of each step may be changed, some steps may be performed repeatedly, some steps may be omitted, or some steps may be added.

FIG. 12 is a flowchart illustrating a method 1200 for generating a reinforcement learning-based container device location recommendation model according to an embodiment of the present disclosure. Method 1200 may be performed by at least one processor (e.g., processor 320) of an information processing system. As shown, the method 1200 may be initiated by setting virtual container data, state data of the virtual container yard, and state data of the virtual reach stacker (S1210). In this case, the virtual container data can be set randomly.

Then, the processor can generate virtual container work data from the set data using a reinforcement learning model (S1220). In one embodiment, virtual container task data may include the task type and device location of the virtual container. For example, the operation type may include receipt of the virtual container, and the device location of the virtual container may include the location at which the virtual container will be stored in the virtual container yard. As another example, the task type may include virtual container rehandling, and the virtual container's device location may include the location at which the virtual container will be rehandled.

In one embodiment, the processor may determine whether the virtual container task data violates a preset prohibition condition. If virtual container task data violates prohibition conditions, virtual container data may be reset randomly. Additionally, virtual container work data may be generated under conditions that limit the device rate of the virtual container yard to an arbitrary value.

Then, the processor may generate a task performance result by transmitting the virtual container task data to the virtual reach stacker (S1230). Finally, the processor can obtain compensation for the results of task performance (S1240). In one embodiment, the processor may update the reinforcement learning model so that the reward obtained by repeating steps S1210 to S1240 described above is maximized.

The flowchart shown in FIG. 12 and the above description are only examples and may be implemented differently in some embodiments. For example, in some embodiments, the order of each step may be changed, some steps may be performed repeatedly, some steps may be omitted, or some steps may be added.

The above-described method may be provided as a computer program stored in a computer-readable recording medium for execution on a computer. The medium may continuously store a computer-executable program, or may temporarily store it for execution or download. In addition, the medium may be a variety of recording or storage means in the form of a single or several pieces of hardware combined. It is not limited to a medium directly connected to a computer system and may be distributed over a network. Examples of media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, And there may be something configured to store program instructions, including ROM, RAM, flash memory, etc. Additionally, examples of other media include recording or storage media managed by app stores that distribute applications, sites or servers that supply or distribute various other software, etc.

The methods, operations, or techniques of this disclosure may be implemented by various means. For example, these techniques may be implemented in hardware, firmware, software, or a combination thereof. Those skilled in the art will understand that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented in electronic hardware, computer software, or combinations of both. To clearly illustrate this interchange of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and design requirements imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementations should not be interpreted as causing a departure from the scope of the present disclosure.

In a hardware implementation, the processing units used to perform the techniques may include one or more ASICs, DSPs, digital signal processing devices (DSPDs), programmable logic devices (PLDs). ), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, electronic devices, and other electronic units designed to perform the functions described in this disclosure. , a computer, or a combination thereof.

Accordingly, the various illustrative logical blocks, modules, and circuits described in connection with this disclosure may be general-purpose processors, DSPs, ASICs, FPGAs or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or It may be implemented or performed as any combination of those designed to perform the functions described in. A general-purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, such as a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other configuration.

For firmware and/or software implementations, techniques include random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), and PROM ( on computer-readable media such as programmable read-only memory (EPROM), electrically erasable PROM (EEPROM), flash memory, compact disc (CD), magnetic or optical data storage devices, etc. It can also be implemented with stored instructions. Instructions may be executable by one or more processors and may cause the processor(s) to perform certain aspects of the functionality described in this disclosure.

Although the above-described embodiments have been described as utilizing aspects of the presently disclosed subject matter in one or more standalone computer systems, the disclosure is not limited thereto and may also be implemented in conjunction with any computing environment, such as a network or distributed computing environment. . Furthermore, aspects of the subject matter of this disclosure may be implemented in multiple processing chips or devices, and storage may be similarly effected across the multiple devices. These devices may include PCs, network servers, and portable devices.

Although the present disclosure has been described in relation to some embodiments in this specification, various modifications and changes may be made without departing from the scope of the present disclosure as can be understood by a person skilled in the art to which the invention pertains. Additionally, such modifications and changes should be considered to fall within the scope of the claims appended hereto.

110: Container device location recommendation system
120: Reinforcement learning model
132: Target container data
134: Status data of actual container yard
136: Status data of actual reach stacker
140: Container task data
150: Reach Stacker

Claims (20)

In a reinforcement learning-based container device location recommendation method performed by at least one processor,
Receiving target container data and status data of an actual container yard;
Receiving status data of an actual reach stacker located in the actual container yard;
Using a reinforcement learning model, generating container task data from the target container data, state data of the actual container yard, and state data of the actual reach stacker; and
Transmitting the container task data to a reach stacker associated with the container task data,
The target container data includes data of the target container retrieved from the status data of the actual container yard and the status data of the actual reach stacker,
The container task data is generated in relation to the current state of the actual reach stacker,
The container task data includes a task type and a device location of the target container,
The operation type includes receipt of the target container and container rehandling,
When the operation type is the warehousing, the device location of the target container includes a location where the target container will be installed in the actual container yard,
When the task type is container rehandling, the device location of the target container includes a location at which the target container will be rehandled,
The reinforcement learning model is,
receive virtual container data, state data of virtual container yards, and state data of virtual reach stackers;
Generate virtual container work data based on the received data,
Transmitting the virtual container work data to the virtual reach stacker,
Learned through a learning method that obtains rewards for task performance results generated by the virtual reach stacker,
The virtual container data includes data of a virtual container retrieved from the state data of the virtual container yard and the state data of the virtual reach stacker,
The virtual container job data is generated in association with the state of the virtual reach stacker.
delete delete delete delete delete In paragraph 1,
A container device location recommendation method, wherein the reinforcement learning model checks whether the virtual container work data violates a preset prohibition condition.
In clause 7,
The virtual container data is set randomly,
The method of recommending a container device location, wherein the virtual container data is randomly reset if the virtual container work data violates the prohibition condition.
According to paragraph 1,
The reinforcement learning model is updated to maximize the reward obtained by repeating the learning method.
According to paragraph 1,
The virtual container work data is,
A container device location recommendation method created under the condition of limiting the device rate of the virtual container yard to an arbitrary value.
In a reinforcement learning-based container device location recommendation model generation method performed by at least one processor,
(a) setting virtual container data, state data of the virtual container yard, and state data of the virtual reach stacker;
(b) using a reinforcement learning model to generate virtual container work data from the set data;
(c) transmitting the virtual container task data to the virtual reach stacker to generate a task performance result; and
(d) comprising obtaining compensation for the results of performing the task,
The virtual container data includes data of a virtual container retrieved from the state data of the virtual container yard and the state data of the virtual reach stacker,
The virtual container task data includes a task type and a device location of the virtual container,
The virtual container work data is generated in association with the state of the virtual reach stacker,
The task type includes receipt of the virtual container and virtual container rehandling,
When the operation type is the warehousing, the device location of the virtual container includes a location in the virtual container yard where the virtual container is to be installed,
When the task type is container rehandling, the location of the virtual container includes a location at which the virtual container will be rehandled.
delete delete delete According to clause 11,
A method for generating a container device location recommendation model, further comprising checking whether the virtual container task data violates a preset prohibition condition.
According to clause 15,
The virtual container data is set randomly,
A method for generating a container device location recommendation model, wherein the virtual container data is randomly reset when the virtual container task data violates the prohibition condition.
According to clause 11,
Updating the reinforcement learning model so that the reward obtained by repeating steps (a), (b), (c), and (d) is maximized.
A method for generating a container device location recommendation model, further comprising:
According to clause 11,
The virtual container work data is,
A method for generating a container device location recommendation model, which is generated under the condition of limiting the device rate of the virtual container yard to an arbitrary value.
A computer program stored in a computer-readable recording medium for executing the method according to any one of claims 1, 7 to 11, and 15 to 18 on a computer.
As an information processing system,
communication module;
Memory; and
At least one processor connected to the memory and configured to execute at least one computer-readable program included in the memory,
The at least one program is,
Receive target container data and status data of the actual container yard,
Receive status data of the actual reach stacker located in the actual container yard,
Using a reinforcement learning model, container task data is generated from the target container data, the state data of the actual container yard, and the state data of the actual reach stacker,
Includes instructions for transmitting the container work data to a reach stacker associated with the container work data,
The target container data includes data of the target container retrieved from the status data of the actual container yard and the status data of the actual reach stacker,
The container task data includes a task type and a device location of the target container,
The operation type includes receipt of the target container and container rehandling,
When the operation type is the warehousing, the device location of the target container includes a location where the target container will be installed in the actual container yard,
When the task type is container rehandling, the device location of the target container includes a location at which the target container will be rehandled,
The container task data is generated in relation to the current state of the actual reach stacker,
The reinforcement learning model is,
receive virtual container data, state data of virtual container yards, and state data of virtual reach stackers;
Generate virtual container work data based on the received data,
Transmitting the virtual container work data to the virtual reach stacker,
Learned through a learning method that obtains rewards for task performance results generated by the virtual reach stacker,
The virtual container data includes data of a virtual container retrieved from the state data of the virtual container yard and the state data of the virtual reach stacker,
The information processing system wherein the virtual container work data is generated in association with the state of the virtual reach stacker.

Images