JP7439459B2 - Machine learning device, conveyance device, image forming device, machine learning method, and program - Google Patents

Description

The present invention relates to a machine learning device, a transport device, an image forming device, a machine learning method, and a program.

2. Description of the Related Art Conventionally, devices equipped with a conveyance device for conveying objects such as paper, film, cloth, etc. are known. As such devices, image forming devices such as printers, copiers, facsimile machines, and multifunction peripherals (MFPs) of these devices are known.

In the field of commercial printing equipment that uses such image forming apparatuses, there is a need to provide output products (printed materials) that meet the needs of users. Therefore, there are various demands on media and image forming apparatuses used for image formation (printing). In order to meet these demands, it is necessary to control image forming apparatuses according to individual situations. However, since it currently relies on manual design, it cannot meet all requests. Manual design has no choice but to create a design that satisfies the worst and typical conditions.

For example, the rotational speed of a roller that conveys paper within a conveying device changes depending on the type of paper and the deterioration of the roller. In order to convey the paper without damage and to satisfy the constraints regarding the deflection of the paper in the conveyance path, it is necessary to appropriately set the rotational speed of the rollers.

Patent Document 1 (Japanese Unexamined Patent Application Publication No. 2014-201409) describes a first conveyance roller, a second conveyance roller adjacent to this on the downstream side, and a first conveyance roller and a second conveyance roller that rotate, respectively. A first drive motor and a second drive motor to be driven, a deflection detection section that detects the amount of deflection of the paper, a torque detection section that detects the amount of torque of the second drive motor, and a first drive motor that detects the amount of deflection of the paper. An image forming apparatus is disclosed that includes a control section that controls a first drive motor and a second drive motor, respectively. The control unit adjusts a relative speed difference in paper transport speed between the first transport roller and the second transport roller when transporting the subsequent paper based on the amount of deflection or torque detected for the preceding paper. .

Additionally, machine learning is attracting attention as computer capabilities have improved in recent years. For example, Patent Document 2 (Japanese Unexamined Patent Publication No. 2017-034844) discloses that by optimizing the current gain parameter in motor control, the response of the motor can be improved, uneven feeding can be improved, and accuracy can be improved. A machine learning device for the purpose is disclosed.

Japanese Patent Application Publication No. 2014-201409 JP2017-034844A

A plurality of rollers (transport means) are installed in a transport device that transports objects such as sheets of paper. For this reason, there are a huge number of combinations of rotational speeds for each roller. Therefore, it is very difficult to optimize the combination of rotational speeds of the rollers (ie, the parameters of the drive means such as motors).

The present disclosure has been made in view of the above-mentioned problems, and an object of the present disclosure is to provide a machine learning device, a transportation device, and an image forming device that can optimize the values of parameters of a driving device that drives a transportation device. , machine learning methods, and programs.

According to an aspect of the present disclosure, a machine learning device includes a state quantity acquisition unit that acquires a state quantity representing a deflection amount or a tension amount of a conveyance target object in a plurality of sections of a conveyance path of a conveyance device. The conveyance device sequentially holds an object to be conveyed by a plurality of conveyance means and conveys the object from upstream to downstream on a conveyance path. The machine learning device is a machine that updates an action value function that represents the value of a set of parameters of each drive means that drives each transport means based on the reward. determining one set from the plurality of sets based on the learning means that performs learning and the updated action value function, and instructing the driving means to drive the conveying means with the parameters of the determined set. and determining means.

Preferably, the state quantity acquisition means further acquires the state quantity when the conveying means is driven based on the selected set of parameters. The reward granting means further grants a reward based on the acquired state amount. The learning means further updates the action value function based on the given reward.

Preferably, the set includes at least one of speed, drive timing, stop timing, shift timing, and drive current value.

Preferably, the state quantity acquisition means acquires the state quantity based on the conveyance speed of the conveyance target object by the conveyance means or the position of the conveyance target object in the conveyance path.

Preferably, it communicates with a simulator that simulates the transport device. The state quantity acquisition means acquires the state quantity based on the output from the simulator.

Preferably, the action value function is a Q-table. The determining means determines one set from the plurality of sets based on the acquired state quantity and Q table.

Preferably, the plurality of conveyance means includes a first conveyance means and a second conveyance means that is the next conveyance means downstream of the first conveyance means. When a setting is made to allow a predetermined amount of deflection in the section between the first conveying means and the second conveying means among the plurality of sections, the reward giving means A positive reward is given when the state quantity in the section between the vehicle and the conveying means represents a deflection amount that is less than or equal to a predetermined deflection amount.

Preferably, the plurality of conveyance means includes a first conveyance means and a second conveyance means that is the next conveyance means downstream of the first conveyance means. If the section between the first conveyance means and the second conveyance means among the plurality of sections is set not to allow deflection of the conveyed object, the reward giving means A positive reward is given when the state quantity in the section between the second transport means and the second transport means represents the amount of tension.

Preferably, the reward giving means gives the reward based on the state quantity and the state of the transport means.

Preferably, a predetermined transport means among the plurality of transport means transports the objects to be transported one by one from a storage means storing a plurality of objects to be transported to the transport path. The reward giving means is such that a state quantity at a position before the rear end of the conveyance target reaches a predetermined conveyance means among the plurality of sections represents the amount of tension, and when the rear end of the conveyance target reaches the predetermined conveyance means. If the predetermined transport means is stopped when passing through the means, a positive reward is given.

Preferably, the plurality of conveyance means includes a first conveyance means and a second conveyance means that is the next conveyance means downstream of the first conveyance means. In the section between the first conveyance means and the second conveyance means among the plurality of sections, the object to be conveyed is deflected and the second conveyance means generates a force on the object to be conveyed in the conveyance direction. If not allowed, the reward giving means gives a positive reward when the state amount in the section between the first conveying means and the second conveying means does not represent either the amount of tension or the amount of deflection. do.

Preferably, the plurality of conveyance means includes a first conveyance means and a second conveyance means that is the next conveyance means downstream of the first conveyance means. If it is possible to speed up the time for the object to be transported through the first transport means by transporting the object in a state where the second transport means pulls the object, the reward giving means A positive reward is given when the means is conveying the conveyed object while pulling it.

Preferably, the plurality of conveyance means includes a first conveyance means and a second conveyance means that is the next conveyance means downstream of the first conveyance means. When the object to be transported is simultaneously transported by the first transport means and the second transport means, a predetermined deflection occurs in the section between the first transport means and the second transport means among the plurality of sections. When the setting is made to allow the amount, the reward giving means gives a positive reward on the condition that the conveyance speed of the first conveyance means is equal to or higher than the conveyance speed of the second conveyance means.

Preferably, the plurality of conveyance means includes a first conveyance means and a second conveyance means that is the next conveyance means downstream of the first conveyance means. When the first conveying means and the second conveying means are simultaneously conveying the conveyed object, in the section between the first conveying means and the second conveying means among the plurality of sections, the conveyed object If the setting is such that the deflection is not allowed, the reward giving means gives a positive reward on the condition that the conveyance speed of the first conveyance means is equal to or lower than the conveyance speed of the second conveyance means.

Preferably, the object to be transported is paper.
Preferably, the object to be transported is cloth.

Preferably, the learning means performs machine learning to update the parameter values of each drive means based on the reward and the physical properties of the conveyed object.

Preferably, the physical property is stiffness.
Preferably, the plurality of conveyance means includes a first conveyance means and a second conveyance means that is the next conveyance means downstream of the first conveyance means. When the object to be transported is simultaneously transported by the first transport means and the second transport means, the remuneration means is configured such that the stiffness of the transport object is equal to or higher than a predetermined value, and the first transport means transports the object. A positive reward is given on the condition that the speed and the transport speed of the second transport means are the same.

Preferably, the plurality of conveyance means includes a first conveyance means and a second conveyance means that is the next conveyance means downstream of the first conveyance means. The object to be transported is simultaneously transported by the first transport means and the second transport means, and a predetermined deflection occurs in the section between the first transport means and the second transport means among the plurality of sections. When the setting is made to allow the amount, the reward giving means determines that the stiffness of the conveyed object is less than a predetermined value and the state quantity in the section between the first conveying means and the second conveying means is a predetermined value. A positive reward is given when the amount of deflection is less than or equal to the amount of deflection.

Preferably, the physical property is basis weight.
Preferably, the plurality of conveyance means includes a first conveyance means and a second conveyance means that is the next conveyance means downstream of the first conveyance means. When the first conveying means and the second conveying means are conveying the conveyed object at the same time, the reward giving means determines that the basis weight of the conveyed object is equal to or greater than a predetermined value and that the first conveying means A positive reward is given on the condition that the transport speed and the transport speed of the second transport means are the same.

Preferably, the plurality of conveyance means includes a first conveyance means and a second conveyance means that is the next conveyance means downstream of the first conveyance means. The object to be transported is simultaneously transported by the first transport means and the second transport means, and a predetermined deflection occurs in the section between the first transport means and the second transport means among the plurality of sections. If the setting is made to allow the amount, the reward giving means determines that the basis weight of the object to be transported is less than a predetermined value and the state quantity in the section between the first transport means and the second transport means is a predetermined value. A positive reward is given when the amount of deflection is less than or equal to the amount of deflection.

Preferably, the conveying means is a pair of rollers.
Preferably, the predetermined amount of deflection is less than the width of the conveyance path in the thickness direction of the object to be conveyed.

Preferably, the predetermined amount of deflection is a predetermined value greater than zero.
Preferably, the state quantity acquisition means acquires the amount of deflection using a simulation model of a mechanical sensor provided in the transport device.

Preferably, the state quantity acquisition means acquires the amount of deflection based on an output from a mechanical sensor provided in the transport device.

Preferably, the state quantity acquisition means acquires the amount of deflection using a simulation model of an optical sensor provided in the transport device.

Preferably, the state quantity acquisition means acquires the amount of deflection based on an output from an optical sensor provided in the transport device.

Preferably, the state quantity acquisition means acquires the length of the object to be transported based on the position of the object to be transported. If the reference length of the object to be transported is longer than the obtained length, the state quantity obtaining means determines the difference between the obtained length and the reference length as the amount of deflection.

Preferably, the state quantity acquisition means acquires the load of the conveyance means using a simulation model of a load detection means provided in the conveyance apparatus. The state quantity acquisition means acquires the amount of tension based on the load.

Preferably, the state quantity acquisition means acquires the amount of tension based on the load detected by the load detection means provided in the conveying device.

Preferably, the state quantity acquisition means acquires the amount of tension using a simulation model of an optical sensor provided in the transport device.

Preferably, the state quantity acquisition means acquires the amount of tension based on an output from an optical sensor provided in the conveyance device.

Preferably, the plurality of conveyance means includes a first conveyance means and a second conveyance means that is the next conveyance means downstream of the first conveyance means. The state quantity acquisition means acquires the length of the object to be transported based on the position of the object to be transported. The state quantity acquisition means determines that when there is no difference between the acquired length and the reference length of the object to be conveyed, and the conveyance speed of the second conveyance means is equal to or higher than the conveyance speed of the first conveyance means, the object to be conveyed is It is considered to be in a pulled state.

Preferably, the machine learning device performs machine learning again when the transport speed of the object to be transported by the transport means is different from the transport speed set during the previous machine learning.

Preferably, the machine learning device transmits a control program for updating that includes parameters as a result of machine learning to a controller that controls the operation of the transport device.

According to another aspect of the present disclosure, a transport device includes the machine learning device described above.
According to yet another aspect of the present disclosure, an image forming apparatus includes the machine learning device described above.

According to still another aspect of the present disclosure, the machine learning method includes the step of acquiring state quantities representing the amount of deflection or the amount of tension of the conveyance target at a plurality of positions on the conveyance path of the conveyance device. The conveyance device sequentially holds an object to be conveyed by a plurality of conveyance means and conveys the object from upstream to downstream on a conveyance path. The machine learning method includes a step of providing a reward based on a state quantity, a step of updating an action value function representing the value of a set of parameters of each driving means that drives each transport means based on the reward, The method further includes the step of determining one set from the plurality of sets based on the updated action value function, and instructing the driving means to drive the conveying means with the parameters of the determined set.

According to still another aspect of the present disclosure, a program for operating a computer causes the processor of the computer to acquire state quantities representing the amount of deflection or the amount of tension of the conveyance target at a plurality of positions on the conveyance path of the conveyance device. Let it run. The conveyance device sequentially holds an object to be conveyed by a plurality of conveyance means and conveys the object from upstream to downstream on a conveyance path. The program includes a step of giving a reward based on the state quantity, a step of updating an action value function representing the value of a set of parameters of each driving means that drives each conveyance means based on the reward, and a step of determining one set from the plurality of sets based on the action value function of the set, and instructing the driving means to drive the conveying means with the parameters of the determined set. ,program.

According to the present disclosure, it is possible to optimize the values of the parameters of the driving means for driving the conveying means.

FIG. 2 is a schematic diagram for explaining an overview of reinforcement learning. FIG. 1 is a diagram showing a learning system according to the present embodiment. 1 is a schematic diagram showing the internal structure of an image forming apparatus. FIG. 2 is a block diagram for explaining an example of the hardware configuration of an image forming apparatus. FIG. 2 is a schematic diagram showing the configuration of a part of the transport device. FIG. 2 is a schematic diagram for explaining data used in Q-learning. It is a schematic diagram for explaining the modification of a conveyance device. FIG. 2 is a diagram showing a typical example of the hardware configuration of a learning device. FIG. 2 is a schematic diagram for explaining the outline of a Q table. 1 is a functional block diagram for explaining the functional configuration of an image forming apparatus 1. FIG. FIG. 2 is a functional block diagram for explaining the configuration of a learning device and the configuration of a simulator. FIG. 3 is a schematic diagram illustrating a state in which a sheet of paper P passes through a portion where the conveyance path has a normal width. FIG. 3 is a schematic diagram showing a state in which paper is bent. FIG. 2 is a schematic diagram illustrating a state in which paper is stretched. FIG. 3 is a schematic diagram illustrating a state in which paper passes through a narrow area. FIG. 3 is a schematic diagram showing a state in which paper P is being supplied to a conveyance path 15 from a paper feed cassette 14 of the image forming apparatus. 3 is a schematic diagram illustrating a state in which printed paper P is discharged to a discharge tray 271 in the image forming apparatus. FIG. It is a flow diagram showing the procedure of Q-learning processing. FIG. 3 is a flowchart for explaining an example of giving compensation regarding the amount of deflection. FIG. 3 is a flow diagram for explaining processing for determining an allowable amount of deflection for each section. It is a flowchart for explaining the example of granting the reward regarding the amount of pull. FIG. 2 is a schematic diagram illustrating a configuration for implementing reinforcement learning within an image forming apparatus.

A system in an embodiment will be described below with reference to the drawings. In the embodiments described below, when referring to the number, amount, etc., the scope of the present disclosure is not necessarily limited to the number, amount, etc. unless otherwise specified. Identical or equivalent parts will be given the same reference numbers, and duplicate descriptions may not be repeated.

In the drawings, some parts are not shown according to the actual size ratio, but are shown with the ratio changed to make the structure clearer, in order to make the structure easier to understand. Note that each modification described below may be selectively combined as appropriate.

Furthermore, in the following description, an image forming apparatus will be used as an example of a device equipped with a transport device as a target object, but the present invention is not limited thereto. Examples of the object include paper, film, cloth, and the like.

Note that examples of the image forming apparatus include a color printer, a monochrome printer, a FAX, and a multi-functional peripheral (MFP).

<A. Machine learning＞
Before describing a specific example of the learning system according to this embodiment, an overview of machine learning used in this example will be briefly described below.

Deep learning and reinforcement learning are known as examples of machine learning. For example, Q learning and TD learning are known as reinforcement learning.

Furthermore, machine learning using deep learning and reinforcement learning is called "deep reinforcement learning (DQN: Deep Q-Network)." Deep reinforcement learning is a method that uses deep learning to express the action value function of reinforcement learning. Deep reinforcement learning is a type of reinforcement learning.

In this embodiment, reinforcement learning will be explained. In particular, Q learning (more specifically, learning using a Q table, which is a type of action value function) will be explained as an example. Note that reinforcement learning is not limited to a configuration using a Q table, and may be deep reinforcement learning. Furthermore, the learning method is not particularly limited as long as it is learning in which an action is selected based on the state and a reward is given based on the selected action.

FIG. 1 is a schematic diagram for explaining an overview of reinforcement learning.
Referring to FIG. 1, an agent observes the state s of the environment at a certain timing. Next, the agent determines action a based on state s and policy π. Policy π is a rule for determining behavior. Optimizing policy π amounts to optimizing action selection. Next, the agent obtains a reward r from the environment based on action a.

Furthermore, the agent repeats the acquisition of the state s, the determination (selection) of the action a based on the policy π, and the acquisition of the reward r based on the determined action a until the agent reaches the final state. That is, the agent repeats the above-described process until the behavior selection (specifically, the parameters) is optimized.

Reinforcement learning typically uses a state value function V(s) expressed by the following equation (1).

V(s)=Σ π(a|s)Q(s,a)...(1)
Specifically, equation (1) is an equation that indicates the sum of all terms for a. In equation (1), π(a|s) represents a stochastic policy. π(a|s) represents the probability of selecting action a in state s. On the other hand, Q(s, a) represents an action value function.

Among reinforcement learning, a method (value-based (value repetition)) that focuses on the behavior value function Q(s, a) is called Q-learning. Further, among reinforcement learning, a method (policy-based (policy repetition)) that focuses on a stochastic policy π(a|s) is called a policy gradient method.

In Q-learning, each time the agent selects action a, the action value function Q(s, a) is updated according to the following equation (2).

Q(s _t , a _t )←Q(s _t , _at )+α{r _t+1 +γmax _a Q(s _t+1 ,a _t+1 )−Q(s _t , _at )} … (2)
Q(s _t , a _t ) represents the expected value of reward obtained by performing action a _t in state s _t . r _t+1 is the immediate reward given for action at at time t+1. α is a learning rate (0<α<1) that determines the speed of learning. γ is a discount rate (0<γ<1). Note that the discount rate is used to prevent the Q table from diverging.

<B. System configuration>
FIG. 2 is a diagram showing a learning system 1000 according to this embodiment.

Referring to FIG. 2, learning system 1000 includes an image forming apparatus 1 used by a user (on the user side, edge side) and an information processing apparatus on the cloud side. In the learning system 1000, Q learning is executed on the cloud. Specifically, the image forming apparatus 1 uploads data detected by the image forming apparatus 1 to the cloud. Thereby, the state of the image forming apparatus 1 is reproduced on the cloud, and Q learning is executed on the cloud. More details are as follows.

On the cloud, one or more information processing devices perform an emulator function, a simulator function, and a reinforcement learning function (AI function). For example, a function as an emulator, a function as a simulator, and a function to perform reinforcement learning are performed by separate information processing devices. The present invention is not limited to this, and the functions as an emulator and a simulator may be executed by one information processing device, and reinforcement learning may be executed by another information processing device. Further, the function as a simulator and the function to perform reinforcement learning may be performed by the same information processing device.

The simulator is a mechanical simulator. The simulator reproduces the operation of the image forming apparatus 1 as seen from the outside.

The simulator simulates the operation of the image forming apparatus 1 using a simulation model corresponding to the image forming apparatus 1. Upon receiving an individual request (customization request) regarding the image forming apparatus 1 from a user, the manufacturer generates a simulation model based on the individual request.

More specifically, the simulation model is generated using a component model provided by an emulator.

The emulator includes models such as fixing device emulator parts, process emulator parts, and conveyance device emulator parts. The emulator reproduces even the internal operations of the image forming apparatus 1. The model of each component is updated according to the status of each device of the image forming apparatus 1.

In the learning system 1000, a simulator and an information processing device (hereinafter also referred to as a “learning device”) that executes reinforcement learning cooperate with each other, and a driving unit (typically, , motor). Specifically, in this example, the learning device determines the parameters in the Q table, which is an example of the action value function Q(s, a), through learning.

More specifically, prior to Q-learning, the image forming apparatus 1, which is a real device, uploads various data detected within the image forming apparatus 1 (also referred to as "sensing data") to the cloud. On the cloud side, the sensing data is used to update the model representing each part in the emulator. More specifically, the model parameters are updated. Additionally, various settings of the simulator are updated using the sensing data. This allows the simulator to better reflect the state of the actual image forming apparatus 1. Note that the simulator includes various parts defined in the emulator (parts whose parameters have been updated using sensing data).

When the Q learning is completed, a program for updating the firmware for the image forming apparatus 1 is generated by the information processing apparatus on the cloud based on the determined parameters. The generated update program is sent to the image forming apparatus 1.

In the image forming apparatus 1, the firmware in the control device of the image forming apparatus 1 is updated by the update program.

<C. Hardware configuration of image forming apparatus>
(c1. Internal structure)
FIG. 3 is a schematic diagram showing the internal structure of the image forming apparatus 1. As shown in FIG. Referring to FIG. 3, the image forming apparatus 1 includes the main body section 10 and the post-processing device 20, as described above.

The main body 10 includes an image forming unit 11, a scanner unit 12, an automatic document transport unit 13, two paper feed cassettes 14, a transport path 15, a media sensor 16, a reverse transport path 17, and an operation panel 34. and a paper feed roller 113. Note that the automatic document feeder unit is also called an ADF (auto document feeder).

The main body 10 further includes a controller 31 that controls the operation of the image forming apparatus 1 . In this example, the main body 10 is a so-called tandem color printer. The main body section 10 executes image formation based on the print settings.

The automatic document transport unit 13 automatically transports the document placed on the document table to the reading position of the document reading section. The scanner unit 12 reads the image of the document conveyed by the automatic document conveyance unit 13 and generates image data. The scanner unit 12 also reads an image of a document placed on a platen by a user without using the automatic document conveyance unit 13, and generates image data. The image data of the document acquired by the scanner unit 12 is stored in a memory (typically, the fixed storage device 32 shown in FIG. 4).

The paper feed cassette 14 stores sheets such as paper P. In the example shown in FIG. 3, the paper feed roller 113 feeds the paper P upward along the conveyance path 15. The paper feed cassette 14 includes a bottom raising plate 142 and a sensor 143. The sensor 143 detects the position of a regulating plate (not shown) in the paper feed cassette, and also detects the size of the paper. Note that sheets other than paper include, for example, envelopes, OHP (overhead projector) films, and cloth.

The conveyance path 15 is used during single-sided printing and double-sided printing. The reversing conveyance path 17 is used during double-sided printing.

The image forming unit 11 forms an image on the paper P fed by the paper feed cassette 14 based on image data generated by the scanner unit 12 or print data acquired from an external device.

The image forming unit 11 includes an intermediate transfer belt 101, a tension roller 102, a drive roller 103, a yellow image forming section 104Y, a magenta image forming section 104M, a cyan image forming section 104C, and a black image forming section. 104K, an image density sensor 105, a primary transfer device 111, a secondary transfer device 115, a registration roller pair 116, and a fixing device 120 including a heating roller 121 and a pressure roller 122. The tension roller 102 and the drive roller 103 hold the intermediate transfer belt 101 and rotate the intermediate transfer belt 101 in the direction A in the figure. The registration roller pair 116 transports the paper P transported by the paper feed roller 113 further downstream.

The media sensor 16 is installed on the conveyance path 15. The media sensor 16 realizes a paper type automatic detection function (a function that automatically detects the paper type). Media sensor 16 is installed between paper feed roller 113 and registration roller pair 116.

The media sensor 16 is, for example, an optical sensor that includes a light emitting section that irradiates light onto a sheet of paper and a light receiving section that receives light reflected by the sheet of paper. An optical sensor serving as the media sensor 16 determines the basis weight of paper from the voltage value of the received light. The media sensor 16 includes a displacement sensor that detects the thickness of the paper, a capacitance sensor that detects the moisture content of the paper, a camera that captures the surface quality of the paper, an ultrasonic sensor that detects the characteristics of the paper, etc. do. Typically, after the paper is set in the paper feed cassette 14, the type of paper is determined by the media sensor 16 when the first sheet of paper is fed from the paper feed cassette 14.

Basis weight information is sent from the media sensor 16 to the controller 31. Thereby, the controller 31 determines the type of paper.

The post-processing device 20 further includes a punching device 220, a side stitching section 250, a saddle stitching section 260, an ejection tray 271, an ejection tray 272, and a lower ejection tray 273.

(c2. Hardware configuration)
FIG. 4 is a block diagram for explaining an example of the hardware configuration of the image forming apparatus 1. As shown in FIG.

Referring to FIG. 4, the main body 10 includes a controller 31, a fixed storage device 32, a short-range wireless IF (Inter Face) 33, a scanner unit 12, an operation panel 34, a paper feed cassette 14, and a media It has a sensor 16, an image forming unit 11, a printer controller 35, a network IF 36, and a wireless IF 37. Each section 11, 12, 14, 16, 32-37 is connected to the controller 31 via a bus 30.

Controller 31 controls the operation of image forming apparatus 1 . The controller 31 includes a CPU (Central Processing Unit) 311, a ROM (Read Only Memory) 312 in which a control program is stored, an S-RAM (Static Random Access Memory) 313 for work, and various settings related to image formation. It has a battery-backed NV-RAM (Non-Volatile RAM: nonvolatile memory) 314 for storing the information, and a clock IC (Integrated Circuit) 315. Each section 311 to 315 is connected via a bus 30. Further, the controller 31 is typically built into the main body 10 as a control base.

The operation panel 34 has keys for performing various inputs and a display section. The operation panel 34 typically includes a touch screen and hardware keys. Note that a touch screen is a device in which a touch panel is superimposed on a display.

The network IF 36 transmits and receives various information to and from external devices connected via the network, including the PC 3, a server (not shown), and another image forming apparatus (not shown).

The printer controller 35 generates a copy image from the print data received by the network IF 36. The image forming unit 11 forms a copy image on paper.

Note that the fixed storage device 32 is typically a hard disk device. The fixed storage device 32 stores various data. Note that the fixed storage device 32 may be a flash memory.

(c3. Conveyance device)
FIG. 5 is a schematic diagram showing the configuration of a part of the transport device. Note that the conveyance device is a mechanism for conveying paper P, which is an object to be conveyed, within the image forming apparatus 1. The transport device includes a plurality of transport units. Each conveyance unit includes a pair of rollers as a conveyance means and a motor as a drive means for driving the pair of rollers. Note that the roller pair typically includes a drive roller and a driven roller that rotates as the drive roller rotates.

Referring to FIG. 5, conveyance device 39 includes a plurality of roller pairs. For example, the conveyance device 39 includes a pair of paper feed rollers, a pair of timing rollers, a pair of fixing rollers, and a pair of paper discharge rollers.

The drive rollers of the paper feed roller pair are configured to be movable by a paper feed clutch. When the paper feed clutch is on, the drive roller of the paper feed roller pair comes into contact with the driven roller. When the paper feed clutch is off, the drive roller of the paper feed roller pair is separated from the driven roller.

Regarding the timing roller pair, the drive roller rotates when the timing clutch is on, and the drive roller stops when the timing clutch is off.

The fixing roller pair is configured to be movable by a fixing clutch (not shown). When the fixing clutch is on, the driving roller of the fixing roller pair comes into contact with the driven roller. When the fixing clutch is off, the driving roller of the fixing roller pair is separated from the driven roller.

The paper ejection roller pair is configured to be movable by a paper ejection clutch (not shown). When the paper ejection clutch is on, the drive roller of the paper ejection roller pair comes into contact with the driven roller. When the paper ejection clutch is off, the driving roller of the paper ejection roller pair is separated from the driven roller.

The paper feed roller and timing roller are rotationally driven by a main motor. The fixing roller is rotationally driven by a fixing motor. The paper ejection roller is rotationally driven by a paper ejection motor. The rotational speed and the like of each motor are instructed by the controller 31.

A plurality of sensors #1, #2, . . . #20 for detecting the position of the paper P are installed on the conveyance path 15. Note that the number of sensors is set to the above number to simplify the drawing, and the number of sensors is not limited to this number. However, it is preferable that the distance between adjacent sensors be equal to or less than the distance (pitch) between sheets P that are continuously conveyed. This is to detect the passage of paper P one by one.

Note that a detection signal from the sensor is sent to the controller 31. The controller 31 determines the position of the paper P based on the output from each sensor.

The operation of the transport device 39 and the output of the sensing results are also reproduced in a simulator on the cloud for Q learning.

FIG. 6 is a schematic diagram for explaining data used in Q-learning.
Referring to FIG. 6, in Q learning, the outputs (on or off) of sensors #1, #2, . . . #20 are set to state s. In addition, in Q learning, the speeds, drive timings, stop timings, speed change timings, and drive current values of the main motor, fixing motor, and paper ejection motor are parameters to be learned. In addition, in Q learning, on/off of the paper feed clutch and the timing clutch are also learning targets and parameters.

Action a corresponds to "determining (selecting) an action with reference to the Q table, and driving each motor with a parameter (specifically, a set of parameters) associated with the determined action." In addition, a reward r can be obtained by action a.

In this example, a reward r is given based on a state quantity (hereinafter referred to as a "paper state quantity") representing the amount of deflection or tension of the paper P, which is the object. Depending on the section of the paper P (section on the conveyance path), it is preferable that the paper P be in a stretched state, that flexing is allowed, or that sagging is not allowed. For this reason, the paper state quantity is acquired in a plurality of sections of the conveyance path, and the reward r is given based on the paper state quantity.

As described above, since the simulator of the image forming apparatus 1 is used in this example, the configuration of the transport device 39 is also reproduced on the simulator. During machine learning, detection of the position of paper P using a sensor is also performed in the simulator.

More specifically, first, the paper state quantity is obtained from the image forming apparatus 1, which is an actual machine, and then the paper state quantity is calculated by a simulator. A reward r is calculated (awarded) for each paper state quantity calculated by the simulator.

Furthermore, the learning device determines the amount of paper condition (the amount of deflection or tension) based on the transport speed of the paper P by the transport device 39 or the position of the paper P in the transport path 15 instead of the position of the paper P in the transport path 15. amount) may be calculated. Note that the conveyance speed is expressed as the product of the rotation speed (rotation speed) of the roller and the radius of the roller.

FIG. 7 is a schematic diagram for explaining a modification of the transport device 39.
Referring to FIG. 7, a transport device 39A is a diagram in which some of the plurality of sensors of the transport device 39 shown in FIG. 5 are replaced with virtual sensors.

In the image forming apparatus 1, which is an actual machine, it would be too costly to arrange a large number of sensors on the conveyance path 15. Therefore, sensors are virtualized. Whether or not the paper P has reached the virtualized sensor is determined based on the position of the paper P and the transport speed. By sequentially updating the position of the paper P, it is determined whether the paper P has reached the sensor position.

<D. Hardware configuration>
FIG. 8 is a diagram showing a typical example of the hardware configuration of the learning device 500. Referring to FIG. 8, learning device 500 includes, as main components, a CPU 581 that executes a program, and a device that volatilely stores data generated by executing the program by CPU 581 or data input via an input device. A ROM 583 that stores data in a non-volatile manner, an HDD 584 that stores data in a non-volatile manner, a display 585, operation keys 586, a communication IF (Interface) 587, and a power supply circuit 588. Each component is interconnected by a data bus.

The power supply circuit 588 is a circuit that steps down the voltage of the commercial power supply received via the outlet and supplies power to each part of the learning device 500.

The communication IF 587 is an interface for communicating with other information processing devices (devices on the cloud or devices on the edge side).

The operation keys 586 are keys (keyboard) used by the user of the learning device 500 to input data to the learning device 500.

Processing in the learning device 500 is realized by each piece of hardware and software executed by the CPU 581.

Each component constituting the learning device 500 shown in the figure is common. Therefore, it can be said that the essential part of the present invention is software stored in the RAM 582, HDD 584, a storage medium, or software that can be downloaded via a network. Note that since the operation of each hardware of the learning device 500 is well known, detailed explanation will not be repeated.

Further, the simulator 800 also has the same hardware configuration as the learning device 500. Therefore, the hardware configuration of the simulator 800 will not be repeatedly described here.

<E. Q table>
FIG. 9 is a schematic diagram for explaining the outline of the Q table.

Referring to FIG. 9, in the Q table 535, state numbers #1 to #1048576 are associated with all combinations ( ²²⁰ combinations) of ON and OFF of each sensor #1 to #20. It is being

Action a is roughly divided into eight groups of actions a1 to a8. Actions a1 to a8 are defined based on eight (=2 ³ ) combinations of paper feed clutch on and off, timing clutch on and off, and paper discharge clutch on and off. For example, action a1 represents a case in which each of the paper feed clutch, timing clutch, and paper discharge clutch is off.

However, the conveyance device 39 also includes clutches other than the above-mentioned clutches. Therefore, in reality, action a can be roughly divided into a larger number of actions.

Action a1 typically includes a plurality of actions a1_1 to a1_n. Note that n is a natural number of 2 or more. Similarly, the action a2 includes n actions as well as each of the other actions a3 to a8, which are composed of a plurality of actions a2_1 to a2_n.

For example, actions a2_1 to a2_n in action a2 include at least the rotational speed, drive timing, stop timing, and shift timing of each of the conveying roller pair with the clutch in the on state and the conveying roller pair without the clutch, and drive current values.

Action a2 will be explained below using an example. Assume that in action a2, there are two pairs of conveying rollers with clutches in the ON state, and one pair of conveying rollers with no clutch. Note that one of the conveyance roller pairs with the clutch in the on state is the paper feed clutch. In this case, three drive means (motors) are driven. Therefore, action a2 is classified based on the drive parameters of the three motors.

There are at least five parameters for one motor: rotational speed, drive timing, stop timing, speed change timing, and drive current value. Assuming that there are 10 setting categories for each parameter, there will be 100,000 (=10 ⁵ ) combinations for one motor.

Therefore, assuming that the parameter settings for the other two motors are the same as above, there are 100,000×100,000×10,000 (=10 ¹⁵ ) combinations of the three motors. In this case, the value of action a2_n is 10 ¹⁵ .

The setting classification of each parameter can be set, for example, ±5, ±10, . . . with respect to the reference value. Note that the values indicated by plus or minus are merely examples, and the values are not limited thereto.

As described above, in the Q table 535, the value of each action is stored in association with each action. Specifically, in the Q table 535, the value of a set of parameters (setting parameters) for driving the motor is shown for each set. In the Q table 535, the value of one set is represented as one numerical value.

Note that the initial value of each numerical value of behavior a in the Q table is set in advance before learning starts.
Next, the selection of action a using the Q table shown in FIG. 9 will be explained. In other words, parameter settings using the Q table will be explained. Furthermore, updating of numerical values (ie, parameters) in the Q table will also be explained.

For example, assume that paper P is transported and state s becomes state #2. In this case, the learning device 500 selects the action a with the highest value from among the actions a1_1 to a1_n, a2_1 to a2_n, . . . , a8_1 to a8_n corresponding to state #2. Specifically, the learning device 500 selects the highest numerical value from among the actions a1_1 to a1_n, a2_1 to a2_n, . . . , a8_1 to a8_n corresponding to state #2.

If the value of action a2_n is 10 ¹⁵ as in the above example, action a with the highest value is selected from among 10 ¹⁵ . However, the learning device 500 does not always select the action a with the highest numerical value, but selects the action a with other numerical values using the ε-greedy method.

When action a is selected, learning device 500 determines the numerical values of various parameters (rotation speed, drive timing, stop timing, shift timing, and drive current value) to drive the transport device within the simulator.

The learning device 500 gives a reward r _t+1 based on the action a (hereinafter referred to as action a _t ) selected in the state s _t . Specifically, a paper state quantity representing the amount of deflection or tension of the paper P is calculated based on the output of the sensor detected based on the result of the action a _t (next state s _t+1 ).

The learning device 500 gives a positive or negative reward based on the calculated paper state amount. Specifically, for example, if action a3_51 is selected, the learning device 500 adds the reward determined based on the paper state quantity to the numerical value of action a3_51 (the numerical value in the table) (or subtracts the value when the reward is negative). ) to update.

By the way, depending on the section of the conveyance path 15, there are sections where deflection is allowed, and there are sections where deflection is not allowed. Furthermore, there are sections of the conveyance path 15 in which it is preferable for the paper P to be in a stretched state. Therefore, the learning device 500 gives a reward in consideration of the section of the conveyance path. An example of giving compensation will be described later.

For example, if the reference length of the paper P is longer than the length calculated based on the position of the paper P, the learning device 500 obtains the difference between the calculated length and the reference length as the amount of deflection. Further, if the length calculated based on the position of the paper P is longer than the reference length of the paper P, the learning device 500 obtains the difference between the calculated length and the reference length as the amount of tension.

Further, the learning device 500 is configured such that there is no difference between the length calculated based on the position of the paper P and the reference length of the paper P, and the conveyance speed of the roller is equal to or higher than the conveyance speed of the roller one position upstream of the roller. If this is the case, it is determined that the paper P is in a stretched state.

Note that the learning device 500 may calculate the amount of deflection of the paper P based on, for example, an imaging result by a camera component model (optical sensor component model) in an emulator. Alternatively, the learning device 500 may measure the amount of deflection of the paper P using, for example, a deflection mechanical sensor component model (actuator component model, etc.) within the emulator.

Further, the learning device 500 may calculate the amount of tension on the paper P by detecting the load applied to the conveyance device using an emulator component of the conveyance device. Alternatively, the learning device 500 may calculate the amount of tension on the paper P based on the imaging results obtained by the camera component model within the emulator.

Note that, as described above, the learning device 500 acquires sensing data from the image forming device 1, which is an actual device. At this time, based on the output from the sensor (see FIG. 5) that detects the position of the paper P, the learning device 500 acquires the amount of deflection or tension from the image forming device 1 using the method described above before starting learning. be able to. Such paper state quantities are reflected when setting the simulator.

<F. Functional configuration>
FIG. 10 is a functional block diagram for explaining the functional configuration of the image forming apparatus 1. As shown in FIG.

Referring to FIG. 10, image forming apparatus 1 includes a controller (control unit) 31, a network IF (Interface) 36, and a transport device 39.

The controller 31 stores firmware. The controller 31 controls the overall operation of the image forming apparatus 1 using firmware or the like. For example, controller 31 controls the rotational speed of the motor.

The transport device 39 includes a plurality of transport units 391_1, 391_2, . . . . Each transport unit 391_1, 391_2, . . . includes a transport section (transport means, roller pair) 398 and a drive section (motor) 399. The transport device 39 operates based on firmware settings. For example, the drive unit 399 is driven according to various parameters (rotational speed, drive timing, stop timing, shift timing, drive current value, etc.) set in firmware.

Network IF 36 is a communication interface for communicating with an external network. The image forming apparatus 1 can communicate with each device (learning device 500, emulator 700, simulator 800) on the cloud 900 through the network IF 36. Specifically, the network IF 36 includes a transmitter 361 and a receiver 362.

The transmitter 361 transmits data to the emulator 700 and the simulator 800 on the cloud 900. Specifically, the transmitter 361 transmits the above-described sensing data to the learning device 500. The transmitter 361 transmits, as sensing data, data obtained when the image forming apparatus 1 is operated for several minutes, for example, to the emulator 700 and the simulator 800. The data includes, in addition to the output of the position detection sensor (see FIG. 5), rotational speed, drive timing, stop timing, shift timing, drive current value, and the like. Further, the data may also include a paper state quantity indicating the amount of deflection or tension of the paper P.

Learning device 500 transmits an update program for updating the firmware of image forming device 1 to image forming device 1 . The update program includes numerical values of various parameters (rotational speed, drive timing, stop timing, shift timing, drive current value, etc.) of the optimal behavior a obtained by Q-learning.

Upon receiving the update program from the learning device 500, the image forming apparatus 1 updates the firmware with the update program.

FIG. 11 is a functional block diagram for explaining the configuration of the learning device 500 and the configuration of the simulator.

Referring to FIG. 11, learning device 500 includes a state observation section 510, a reward giving section 520, a learning section 530, a decision making section 540, an update program creation section 550, and an update program transmission section 560. Equipped with. The state observation unit 510 includes a state quantity acquisition unit 515. The learning unit 530 has a Q table 535. In the Q table 535, a state s and an action _at are associated. Further, a set of parameters (a set of a plurality of parameters) is associated with the action _at .

The simulator 800 has an image forming apparatus model 805 that models the image forming apparatus 1 for simulation. The image forming apparatus model 805 includes a controller model 810 that models the controller 31 for simulation, and a transport device model 820 that models the transport device 39 for simulation.

The transport device model 820 includes transport unit models 821_1, 821_2, . . . that are modeled for simulation of a plurality of transport units 391_1, 391_2, . Each transport unit model includes a transport unit model 8212 that models the transport unit 398 (roller pair), and a drive unit model 8214 that models the drive unit 399.

The transport device model 820 further includes sensor models 825_1, 825_2, . . . which model a plurality of sensors (see FIG. 5) for simulation.

The state observation unit 510 acquires the state s (output data from the simulator 800) from the simulator 800. In the state s, in addition to the ON or OFF output from the sensor, the various data described above are acquired.

The state quantity acquisition unit 515 of the state observation unit 510 acquires paper state quantities representing the amount of deflection or tension of the paper P in a plurality of sections of the conveyance path of the conveyance device (conveyance device model). Typically, the state quantity acquisition unit 515 acquires the paper state quantity based on the transport speed of the paper P by the transport unit model 8212 (roller pair, transport means) or the position of the paper P in the transport path 15. .

The reward giving unit 520 gives a reward r based on the state variables (including the paper state amount) from the state observation unit. Typically, the reward giving unit 520 gives the reward r based on the paper state quantity and the state of the transport unit model 8212. An example of granting the reward r will be described later.

The learning unit 530 updates the numerical value (that is, the value) of the corresponding action a in the Q table based on the reward r provided. Specifically, the learning unit 530 uses the obtained reward r to create a Q table 535 that represents the value of a set of parameters of each drive unit model 8214 (motor, drive means) that drives each transport unit model 8212 for each set. Perform machine learning to update based on the information. The set of parameters includes at least one of the speed, drive timing, stop timing, shift timing, and drive current value of each drive unit model 8214.

The decision making unit 540 determines one set from the plurality of sets based on the updated Q table 535, and instructs the drive unit model 8214 to drive the transport unit model 8212 with the parameters of the selected set. Instruct against.

Specifically, the decision making unit 540 refers to the Q table 535 and selects the action a with the highest numerical value (value) from among the plurality of actions a corresponding to the state s. Note that the decision making unit 540 uses the ε-greedy method to select actions a other than the action a with the highest numerical value. The decision making unit 540 instructs the drive unit model 8214 to drive the transport unit model 8212 with the set (set of parameters) associated with the selected action a.

The learning device 500 repeats the above-described series of processes until the final state is reached. That is, the state quantity acquisition unit 515 further acquires the state s when the transport unit model 8212 is driven based on the parameters of the selected set (the set of parameters associated with the selected action a). The reward giving unit 520 further gives a reward r based on the obtained paper state amount. The learning unit 530 further updates the Q table based on the awarded reward. Note that the "final state" includes, for example, a case where the values of parameters instructed to the simulator 800 become constant.

As described above, the learning device 500 communicates with the simulator 800 that simulates the transport device. The state quantity acquisition unit 515 acquires the state s based on the output from the simulator 800. The reward giving unit 520 gives a reward r based on the state s. The learning unit 530 performs machine learning to update the Q table 535, which represents the value of a set of parameters of each drive unit model 8214 that drives each transport unit model 8212, for each set, based on the reward r.

The decision making unit 540 determines one set from a plurality of sets (sets of parameters) based on the updated Q-table, and causes the driving unit to drive the transport unit model 8212 with the parameters of the determined set. Instruct model 8214. Specifically, the decision making unit 540 determines one set from a plurality of sets (parameter sets) based on the acquired state s and the Q table 535, and uses the parameters of the determined set to create the transport unit model 8212. The driver model 8214 is instructed to drive the .

Again, with reference to FIG. 9, the parameters instructed by the decision making unit 540 to the simulator 800 will be explained using a specific example.

In a certain situation, it is assumed that the state s (specifically, state s _t ) of the image forming apparatus model 805 (specifically, the transport device model 820) that the state observation unit 510 acquires from the simulator 800 is state #2. In this case, the reward granting unit 520 grants reward r based on state #2.

The learning unit 530 updates the value of the most recently instructed action a to the simulator 800 based on the reward r. That is, the learning unit 530 updates the numerical values in the columns corresponding to the state #2 and the action a in the Q table 535.

For example, if the most recent action is action a2_2, the learning unit 530 updates the numerical value of action a2_2 in the numerical value group (row) of state #2. That is, the learning unit 530 updates one numerical value where the row of state #2 and the column of action a2_2 intersect.

The decision making unit 540 determines (selects) the next action a _t based on the current state s _t and the current Q table 535 (if updated, the updated Q table 535). Further, the decision making unit 540 notifies the simulator 800 of the parameters associated with the determined action _at .

For example, the decision making unit 540 typically selects the action with the highest numerical value (ie, the highest value) in state #2. Note that, as described above, the ε-greedy method is used to impart randomness to the selection of action a. That is, the decision making unit 540 also executes a process in which the action a with the highest numerical value is intentionally not selected.

When an instruction is given from the decision making unit 540, the drive unit model 8214 of the simulator 800 drives the transport unit model 8212 using the instructed parameters. As a result, the state observation unit 510 observes the next state s _t+1 . Furthermore, a reward r _t+1 based on the action a _t is obtained. Specifically, the reward granting unit 520 grants a reward r _t+1 for the action a _t .

Thereafter, the above-described awarding of rewards, updating of Q table 535, and notification of parameters to simulator 800 are repeated.

According to the above configuration, it is possible to optimize the parameter values of the rollers that drive the image forming apparatus 1, which is an actual machine. Further, as described above, the learning device 500 transmits an update program for updating the firmware of the image forming apparatus 1 to the image forming apparatus 1. Therefore, the transport system of the image forming apparatus 1 can be operated with suitable settings.

<G. Example of remuneration based on paper condition>
(g1. 1st example)
FIG. 12 is a schematic diagram showing a state in which the paper P passes through a portion of the conveyance path 15 where the width W is normal. Note that "width" means a gap in the thickness direction of the paper P.

Referring to FIG. 12, the conveying device 39 includes a roller pair 401 and a roller pair 402 which is the next conveying means downstream of the roller pair 401. The roller pair 401 includes a driving roller 4011 and a driven roller 4012. The roller pair 402 includes a driving roller 4021 and a driven roller 4022. The paper P is conveyed in the direction of the arrow (downstream direction).

FIG. 13 is a schematic diagram showing a state in which the paper P is bent.
Referring to FIG. 13, paper P is bent between roller pair 401 and roller pair 402. For example, when the rotational speed of the upstream roller pair 401 is faster than the rotational speed of the downstream roller pair 402, the paper P is bent in this manner.

FIG. 14 is a schematic diagram showing a state in which the paper P is pulled.
Referring to FIG. 14, paper P is being pulled between roller pair 401 and roller pair 402. For example, when the rotational speed of the downstream roller pair 402 is faster than the rotational speed of the upstream roller pair 401, the paper P is pulled in this manner.

In this example, settings are made to allow a predetermined amount of deflection in the section between the roller pair 401 and the roller pair 402. In this case, the reward giving unit 520 corrects the paper condition amount (deflection amount or tension amount) in the section between the roller pair 401 and the roller pair 402 when the amount of deflection is equal to or less than the predetermined deflection amount. will be given a reward.

In this example, the predetermined amount of deflection is a value less than the width of the conveyance path 15 in the thickness direction of the paper P. Further, the predetermined amount of deflection is a predetermined value in which the amount of deflection is greater than zero.

Note that the setting for allowing a predetermined amount of deflection at the location is registered in advance in the reward giving unit 520. In the following, it is assumed that the settings for the paper state quantity (the amount of deflection or the amount of tension) are registered in advance in the reward giving unit 520.

(g2. Second example)
FIG. 15 is a schematic diagram showing a state in which the paper P passes through a narrow area.

Referring to FIG. 15, the conveying device 39 includes a pair of rollers 403 and a pair of rollers 404 that is the next conveying means downstream of the pair of rollers 403. The roller pair 403 includes a driving roller 4031 and a driven roller 4032. The roller pair 404 includes a driving roller 4041 and a driven roller 4042. The paper P is conveyed in the direction of the arrow (downstream direction).

Since the width W of the conveyance path 15 is narrow, a setting is made in which the sheet P is not allowed to bend in the section between the roller pair 403 and the roller pair 404. In this case, the reward giving unit 520 gives a positive reward when the paper state amount in the section between the roller pair 430 and the roller pair 440 represents the amount of tension.

(g3. Third example)
FIG. 16 is a schematic diagram showing a state in which paper P is being supplied to the conveyance path 15 from the paper feed cassette 14 of the image forming apparatus 1.

Referring to FIG. 16, paper feed roller 113 and roller pair 406 transport sheets P one by one from paper feed cassette 14 storing a plurality of sheets P to transport path 15.

The reward giving unit 520 determines that the paper state amount at a position before the trailing edge of the paper P reaches the paper feed roller 113 represents the amount of tension, and that when the trailing edge of the paper P passes the paper feed roller 113 If the paper feed roller 113 is stopped, a positive reward is given.

(g4. Fourth example)
FIG. 17 is a schematic diagram showing a state in which printed paper P is discharged to the discharge tray 271 in the image forming apparatus 1.

Referring to FIG. 17, the conveying device 39 includes a pair of rollers 407 and a pair of rollers 408 that is the next conveying means downstream of the pair of rollers 407. The roller pair 407 includes a driving roller 4071 and a driven roller 4072. The roller pair 408 includes a driving roller 4081 and a driven roller 4082. The paper P is conveyed (discharged) in the direction of the arrow (downstream direction).

When discharging the paper P to the discharge tray 271, it is possible to speed up the time for the paper P to pass through the roller pair 407 by conveying the paper P while being pulled by the roller pair 408. Therefore, in this case, a positive reward is given when the reward giving unit 520 and the pair of rollers 408 are conveying the paper P in a stretched state.

(g5. Fifth example)
The paper P is sent from the registration roller pair 116 to the nip area of the drive roller 103 and the secondary transfer device (secondary transfer roller) 115, and the image is transferred thereto (see FIG. 13). Further, the image is fixed onto the paper P by a fixing device 120 including a heating roller 121 and a pressure roller 122.

Between the registration roller pair 116, the drive roller 103, and the secondary transfer device 115, it is necessary that the paper P be neither bent nor stretched in order to form a highly accurate image.

Therefore, the reward giving unit 520 gives a positive reward when the paper state amount in the section between the registration roller pair 116 and the secondary transfer device 115 does not represent either the amount of tension or the amount of deflection.

In this way, if the deflection of the paper P and the generation of force on the paper P in the transport direction are not allowed in the section between the two roller pairs (or rollers), the reward giving unit 520 A positive reward is given when the paper state amount in the section between the side roller and the downstream roller pair (the roller pair next to the upstream roller pair) does not represent either the amount of tension or the amount of deflection. do.

(g6. 6th example)
When paper P is conveyed simultaneously by an upstream roller pair and a downstream roller pair (the next roller pair after the upstream roller pair), there is a gap between the upstream roller pair and the downstream roller pair. When the setting is made to allow a predetermined amount of deflection in the section of will be given a reward.

(g7. Seventh example)
When paper P is conveyed simultaneously by an upstream roller pair and a downstream roller pair (the next roller pair after the upstream roller pair), there is a gap between the upstream roller pair and the downstream roller pair. If the setting is such that the paper P is not allowed to bend in the section, the reward giving unit 520 sets the condition that the conveyance speed of the upstream roller pair is equal to or lower than the conveyance speed of the downstream roller pair. , giving positive rewards.

<G. Control structure>
FIG. 18 is a flow diagram showing the procedure of Q learning processing.

Referring to FIG. 18, in step S1, the learning device 500 refers to the acquired state s _t and the current Q table 535 to determine the next action a _t . Specifically, the decision making unit 540 (see FIG. 11) selects the action a _t based on the numerical values in the Q table, and notifies the simulator 800 of the parameters associated with the selected action a _t .

In step S2, the simulator 800 acts based on the _action at determined in step S1. Specifically, the simulator 800 drives the drive unit model 8214 using the parameters notified from the decision making unit 540.

In step S3, the state observation unit 510 acquires the state s _t+1 from the simulator 800 when the drive unit model 8214 is driven using the parameters notified in step S2.

In step S4, the reward giving unit 520 gives a reward r _t+1 to the selected action a based on the paper state quantities representing the amount of deflection and the amount of tension. In step S5, the learning unit 530 updates the Q table 535 based on the granted reward r _t+1 . Specifically, the learning unit 530 updates the value of the selected action a _t by giving the reward r _t+1 .

FIG. 19 is a flowchart for explaining an example of giving compensation regarding the amount of deflection.
In step S11, in the learning device 500, a target value for the amount of deflection for each section is set in advance. In step S12, the learning device 500 measures the amount of deflection based on the state s acquired from the simulator 800. Note that the deflection amount itself may be transmitted from the simulator 800.

In step S13, the learning device 500 compares the target value and the measured value. If the measured value is less than or equal to zero, in step S14, the reward giving unit 520 gives, for example, a reward of "-1" to the selected action. If the measured value is greater than zero and less than or equal to the target value, in step S15, the reward giving unit 520 gives, for example, a reward of "+1" to the selected action. If the measured value is larger than the target value, in step S16, the reward giving unit 520 gives, for example, a reward of "-1" to the selected action.

FIG. 20 is a flow diagram for explaining processing for determining the allowable amount of deflection for each section. Note that to simplify the explanation, an example will be described in which there are three sections.

Referring to FIG. 20, in step S21, sections are set in advance. In step S22, if the paper P is warped, the learning device 500 determines the section where the warp occurs based on the state s.

When the section where the deflection occurs is section A, the allowable amount of deflection is set to 3 mm in step S23. In section A, the learning device 500 provides a reward by comparing the amount of deflection and the allowable amount (3 mm). Similarly, if the section where the deflection occurs is section B, the allowable amount of deflection is set to 5 mm in step S24. In section B, the learning device 500 provides a reward by comparing the amount of deflection and the allowable amount (5 mm). Further, when the section where the deflection occurs is section C, the allowable amount of deflection is set to 2 mm in step S25. In section C, the learning device 500 provides a reward by comparing the amount of deflection and the allowable amount (2 mm).

FIG. 21 is a flowchart illustrating an example of awarding rewards related to the amount of pull.
In step S31, in the learning device 500, a target value of the amount of tension for each section is set in advance. In step S32, the learning device 500 measures the amount of tension based on the state s acquired from the simulator 800. Note that the tension amount itself may be transmitted from the simulator 800.

In step S33, the learning device 500 compares the target value and the measured value. If the measured value is smaller than the target value, in step S34, the reward giving unit 520 gives, for example, a reward of "-1" to the selected action. If the measured value is smaller than zero and greater than or equal to the target value, in step S35, the reward giving unit 520 gives, for example, a reward of "+1" to the selected action. If the measured value is zero or more, in step S16, the reward giving unit 520 gives, for example, a reward of "-1" to the selected action.

<I. Modified example>
(i1. Learning considering physical properties)
The learning unit 530 may update the parameter values of each motor based on the reward and the physical properties of the paper P. That is, the learning device 500 may be configured to perform machine learning by further considering the physical properties of the paper P. In this case, the Q table 535 may be configured as a table that takes physical properties into consideration. Examples of physical properties include stiffness and basis weight. By considering physical properties, it becomes possible to set more optimal parameters.

A case in which stiffness is considered as an example of physical property will be explained as follows.
When the upstream roller pair and the downstream roller pair (the next roller pair after the upstream roller pair) are simultaneously transporting the paper P, the reward giving unit 520 determines that the stiffness of the paper P is equal to or greater than a predetermined value. A positive reward is given on the condition that the transport speed of the upstream roller pair and the transport speed of the downstream roller pair are the same.

Further, the object to be conveyed is simultaneously conveyed by an upstream roller pair and a downstream roller pair (the next roller pair after the upstream roller pair), and the upstream roller pair and the downstream roller pair If the setting is made to allow a predetermined amount of deflection in the section between A positive reward is given when the paper state quantity in the section between 2 and 3 represents a deflection amount that is less than or equal to a predetermined deflection amount.

The case where basis weight is considered as an example of physical properties is as follows.
When the upstream roller pair and the downstream roller pair (the next roller pair after the upstream roller pair) are conveying the paper P at the same time, the reward giving unit 520 determines whether the basis weight of the paper P is greater than or equal to a predetermined value. , and a positive reward is given on the condition that the conveyance speed of the upstream roller pair and the conveyance speed of the downstream roller pair are the same.

Further, the object to be conveyed is simultaneously conveyed by an upstream roller pair and a downstream roller pair (the next roller pair after the upstream roller pair), and the upstream roller pair and the downstream roller pair If the setting is made to allow a predetermined amount of deflection in the section between A positive reward is given when the paper state quantity in the section between .

(i2. Relearning)
After learning, when the user is using the actual image forming apparatus 1, if the conveyance speed of the paper P of any pair of rollers differs from the conveyance speed set during the previous learning, Preferably, the learning device 500 is configured to perform machine learning again.

For example, the learning device 500 is configured to perform machine learning again when the conveyance speed of the paper P in the roller pair exceeds, exceeds, or falls below a reference value from the conveyance speed set during the previous learning. It is preferable to do so.

Alternatively, the learning device can be configured to perform machine learning again when the conveyance speed of the paper P in the roller pair increases or decreases by more than a standard percentage from the conveyance speed set during the previous learning. 500 is preferred.

(i3. Machine learning within image forming apparatus 1)
FIG. 22 is a schematic diagram for explaining a configuration for implementing reinforcement learning within the image forming apparatus 1.

Referring to FIG. 22, image forming apparatus 1 includes an emulator 700, a simulator 800, and a learning device 500. According to such a configuration, there is no need for the information processing device (specifically, the server) on the cloud 900 to perform reinforcement learning.

Note that the above-mentioned reinforcement learning may be performed in the conveyance device instead of the image forming device. That is, reinforcement learning may be performed within a device that transports an object, regardless of whether it has an image forming function or not.

Moreover, the learning method described above can also be provided by a program. The information processing device functions as the learning device 500 by executing the program.

The embodiments disclosed this time should be considered to be illustrative in all respects and not restrictive. The scope of the present invention is indicated by the claims rather than the above description, and it is intended that all changes within the meaning and range equivalent to the claims are included.

1 image forming apparatus, 10 main unit, 11 image forming unit, 12 scanner unit, 13 automatic document transport unit, 14 paper feed cassette, 15 transport path, 16 media sensor, 17 reversing transport path, 20 post-processing device, 30 bus, 31 controller, 32 fixed storage device, 34 operation panel, 35 printer controller, 39, 39A conveyance device, 101 intermediate transfer belt, 102 tension roller, 103, 4011, 4021, 4031, 4041, 4071, 4081 drive roller, 104C, 104K , 104M, 104Y image forming section, 105 image density sensor, 111 primary transfer device, 115 secondary transfer device, 113 paper feed roller, 116 registration roller pair, 120 fixing device, 121 heating roller, 122 pressure roller, 142 bottom raiser board, 143 sensor, 220 punch processing device, 250 side stitching processing section, 260 saddle stitching processing section, 271, 272, 273 discharge tray, 361 transmitting section, 362 receiving section, 391 transport unit, 398 transport section, 399 drive section, 401, 402, 403, 404, 406, 407, 408, 430, 440 roller pair, 500 learning device, 510 state observation unit, 515 state quantity acquisition unit, 520 reward provision unit, 530 learning unit, 535 Q table, 540 intention determination unit, 550 update program creation unit, 560 update program transmission unit, 582 RAM, 583 ROM, 585 display, 586 operation keys, 588 power supply circuit, 700 emulator, 800 simulator, 805 image forming device model, 810 controller model, 820 transport device model, 821 transport unit model, 825 sensor model, 900 cloud, 1000 learning system, 4012, 4022, 4032, 4042, 4072, 4082 driven roller, 8212 transport unit model, 8214 drive unit model, P paper, W width .

Claims (58)

The conveyance apparatus includes a state quantity acquisition means for acquiring a state quantity representing the amount of deflection or tension of the conveyance object in a plurality of sections of the conveyance path of the conveyance apparatus, and the conveyance apparatus sequentially conveys the conveyance target object by the plurality of conveyance means. sandwiching and transporting the object to be transported from upstream to downstream of the transport path,
Reward granting means for granting a reward based on the state quantity;
Learning means that performs machine learning to update an action value function that represents the value of a set of parameters of each driving means that drives each of the transport means, based on the reward;
determining one of the plurality of sets based on the updated action value function, and instructing the driving means to drive the conveying means with the parameters of the determined set; further comprising means;
The state quantity acquisition means acquires the state quantity based on the conveyance speed of the conveyance target object by the conveyance means or the position of the conveyance target object in the conveyance path,
The plurality of conveyance means includes a first conveyance means and a second conveyance means that is the next conveyance means downstream of the first conveyance means,
When a setting is made to allow a predetermined amount of deflection in the section between the first conveying means and the second conveying means among the plurality of sections, the reward giving means A machine learning device that provides a positive reward when the state quantity in the section between the transport means and the second transport means represents the amount of deflection that is equal to or less than the predetermined amount of deflection. The conveying device includes a state quantity acquisition means for acquiring a state quantity representing the amount of deflection or tension of the conveyed object in a plurality of sections of the conveyance path of the conveyance device, and the conveyance device sequentially conveys the conveyed object by the plurality of conveyance means. sandwiching and transporting the object to be transported from upstream to downstream of the transport path,
Reward granting means for granting a reward based on the state quantity;
Learning means that performs machine learning to update an action value function representing the value of a set of parameters of each driving means that drives each of the conveying means for each set based on the reward;
determining one of the plurality of sets based on the updated action value function, and instructing the driving means to drive the conveying means with the parameters of the determined set; further comprising means;
The state quantity acquisition means acquires the state quantity based on the conveyance speed of the conveyance target object by the conveyance means or the position of the conveyance target object in the conveyance path,
The plurality of conveyance means includes a first conveyance means and a second conveyance means that is the next conveyance means downstream of the first conveyance means,
In a section between the first conveying means and the second conveying means among the plurality of sections, when a setting is made in which the deflection of the conveyed object is not allowed, the reward giving means A machine learning device that provides a positive reward when the state amount in the section between the first conveyance means and the second conveyance means represents the amount of tension . The conveyance apparatus includes a state quantity acquisition means for acquiring a state quantity representing the amount of deflection or tension of the conveyance object in a plurality of sections of the conveyance path of the conveyance apparatus, and the conveyance apparatus sequentially conveys the conveyance target object by the plurality of conveyance means. sandwiching and transporting the object to be transported from upstream to downstream of the transport path,
Reward granting means for granting a reward based on the state quantity;
Learning means that performs machine learning to update an action value function that represents the value of a set of parameters of each driving means that drives each of the transport means, based on the reward;
determining one of the plurality of sets based on the updated action value function, and instructing the driving means to drive the conveying means with the parameters of the determined set; further comprising means;
The state quantity acquisition means acquires the state quantity based on the conveyance speed of the conveyance target object by the conveyance means or the position of the conveyance target object in the conveyance path,
The reward granting means grants the reward based on the state quantity and the state of the transport means,
A predetermined transport means among the plurality of transport means transports the transport objects one by one from a storage means storing a plurality of transport objects to the transport path,
The reward giving means is configured such that the state quantity at a position before the rear end of the conveyance object reaches the predetermined conveyance means among the plurality of sections represents the pulling amount, and A machine learning device that provides a positive reward when the predetermined transport means is stopped when the rear end of the vehicle passes through the predetermined transport means . The conveyance apparatus includes a state quantity acquisition means for acquiring a state quantity representing the amount of deflection or tension of the conveyance object in a plurality of sections of the conveyance path of the conveyance apparatus, and the conveyance apparatus sequentially conveys the conveyance target object by the plurality of conveyance means. sandwiching and transporting the object to be transported from upstream to downstream of the transport path,
Reward granting means for granting a reward based on the state quantity;
Learning means that performs machine learning to update an action value function that represents the value of a set of parameters of each driving means that drives each of the transport means, based on the reward;
determining one of the plurality of sets based on the updated action value function, and instructing the driving means to drive the conveying means with the parameters of the determined set; further comprising means;
The state quantity acquisition means acquires the state quantity based on the conveyance speed of the conveyance target object by the conveyance means or the position of the conveyance target object in the conveyance path,
The reward granting means grants the reward based on the state quantity and the state of the transport means,
The plurality of conveyance means includes a first conveyance means and a second conveyance means that is the next conveyance means downstream of the first conveyance means,
In the section between the first conveying means and the second conveying means among the plurality of sections, the object to be conveyed is deflected, and the object to be conveyed in the conveying direction in the second conveying means is If the generation of force is not permitted, the reward giving means determines whether the state quantity in the section between the first conveyance means and the second conveyance means is equal to the amount of tension or the amount of deflection. A machine learning device that gives positive rewards when the user does not represent the same . The conveying device includes a state quantity acquisition means for acquiring a state quantity representing the amount of deflection or tension of the conveyed object in a plurality of sections of the conveyance path of the conveyance device, and the conveyance device sequentially conveys the conveyed object by the plurality of conveyance means. sandwiching and transporting the object to be transported from upstream to downstream of the transport path,
Reward granting means for granting a reward based on the state quantity;
Learning means that performs machine learning to update an action value function representing the value of a set of parameters of each driving means that drives each of the conveying means for each set based on the reward;
determining one of the plurality of sets based on the updated action value function, and instructing the driving means to drive the conveying means with the parameters of the determined set; further comprising means;
The state quantity acquisition means acquires the state quantity based on the conveyance speed of the conveyance target object by the conveyance means or the position of the conveyance target object in the conveyance path,
The reward granting means grants the reward based on the state quantity and the state of the transport means,
The plurality of conveyance means includes a first conveyance means and a second conveyance means that is the next conveyance means downstream of the first conveyance means,
In the case where it is possible to speed up the time for the object to be transported through the first transport means by transporting the object in a pulled state by the second transport means, the reward granting means , a machine learning device that gives a positive reward when the second conveying means conveys the conveyed object in a pulled state . The conveyance apparatus includes a state quantity acquisition means for acquiring a state quantity representing the amount of deflection or tension of the conveyance object in a plurality of sections of the conveyance path of the conveyance apparatus, and the conveyance apparatus sequentially conveys the conveyance target object by the plurality of conveyance means. sandwiching and transporting the object to be transported from upstream to downstream of the transport path,
Reward granting means for granting a reward based on the state quantity;
Learning means that performs machine learning to update an action value function that represents the value of a set of parameters of each driving means that drives each of the transport means, based on the reward;
determining one of the plurality of sets based on the updated action value function, and instructing the driving means to drive the conveying means with the parameters of the determined set; further comprising means;
The state quantity acquisition means acquires the state quantity based on the conveyance speed of the conveyance target object by the conveyance means or the position of the conveyance target object in the conveyance path,
The reward giving means gives the reward based on the state quantity and the state of the transport means,
The plurality of conveyance means includes a first conveyance means and a second conveyance means that is the next conveyance means downstream of the first conveyance means,
When the first conveyance means and the second conveyance means are conveying the object at the same time, the distance between the first conveyance means and the second conveyance means among the plurality of sections is When the setting is made to allow a predetermined amount of deflection in the section, the remuneration granting means provides, on the condition that the conveyance speed of the first conveyance means is equal to or higher than the conveyance speed of the second conveyance means, A machine learning device that gives positive rewards . The conveyance apparatus includes a state quantity acquisition means for acquiring a state quantity representing the amount of deflection or tension of the conveyance object in a plurality of sections of the conveyance path of the conveyance apparatus, and the conveyance apparatus sequentially conveys the conveyance target object by the plurality of conveyance means. sandwiching and transporting the object to be transported from upstream to downstream of the transport path,
Reward granting means for granting a reward based on the state quantity;
Learning means that performs machine learning to update an action value function that represents the value of a set of parameters of each driving means that drives each of the transport means, based on the reward;
determining one of the plurality of sets based on the updated action value function, and instructing the driving means to drive the conveying means with the parameters of the determined set; further comprising means;
The state quantity acquisition means acquires the state quantity based on the conveyance speed of the conveyance target object by the conveyance means or the position of the conveyance target object in the conveyance path,
The reward giving means gives the reward based on the state quantity and the state of the transport means,
The plurality of conveyance means includes a first conveyance means and a second conveyance means that is the next conveyance means downstream of the first conveyance means,
When the first conveyance means and the second conveyance means are conveying the object at the same time, the distance between the first conveyance means and the second conveyance means among the plurality of sections is If a setting is made in which the deflection of the conveyed object is not allowed in the section, the remuneration granting means sets the condition that the conveying speed of the first conveying means is equal to or less than the conveying speed of the second conveying means. A machine learning device that gives positive rewards to people . A machine learning device,
The conveyance apparatus includes a state quantity acquisition means for acquiring a state quantity representing the amount of deflection or tension of the conveyance object in a plurality of sections of the conveyance path of the conveyance apparatus, and the conveyance apparatus sequentially conveys the conveyance target object by the plurality of conveyance means. sandwiching and transporting the object to be transported from upstream to downstream of the transport path,
Reward granting means for granting a reward based on the state quantity;
Learning means that performs machine learning to update an action value function that represents the value of a set of parameters of each driving means that drives each of the transport means, based on the reward;
determining one of the plurality of sets based on the updated action value function, and instructing the driving means to drive the conveying means with the parameters of the determined set; further comprising means;
The machine learning device communicates with a simulator that simulates the transport device,
The state quantity acquisition means acquires the state quantity based on the output from the simulator,
The plurality of conveyance means includes a first conveyance means and a second conveyance means that is the next conveyance means downstream of the first conveyance means,
When a setting is made to allow a predetermined amount of deflection in the section between the first conveying means and the second conveying means among the plurality of sections, the reward giving means A machine learning device that provides a positive reward when the state quantity in the section between the transport means and the second transport means represents the amount of deflection that is equal to or less than the predetermined amount of deflection. A machine learning device,
The conveyance apparatus includes a state quantity acquisition means for acquiring a state quantity representing the amount of deflection or tension of the conveyance object in a plurality of sections of the conveyance path of the conveyance apparatus, and the conveyance apparatus sequentially conveys the conveyance target object by the plurality of conveyance means. sandwiching and transporting the object to be transported from upstream to downstream of the transport path,
Reward granting means for granting a reward based on the state quantity;
Learning means that performs machine learning to update an action value function that represents the value of a set of parameters of each driving means that drives each of the transport means, based on the reward;
determining one of the plurality of sets based on the updated action value function, and instructing the driving means to drive the conveying means with the parameters of the determined set; further comprising means;
The machine learning device communicates with a simulator that simulates the transport device,
The state quantity acquisition means acquires the state quantity based on the output from the simulator,
The plurality of conveyance means includes a first conveyance means and a second conveyance means that is the next conveyance means downstream of the first conveyance means,
In a section between the first conveying means and the second conveying means among the plurality of sections, when a setting is made in which the deflection of the conveyed object is not allowed, the reward giving means A machine learning device that provides a positive reward when the state amount in the section between the first conveyance means and the second conveyance means represents the amount of tension . A machine learning device,
The conveyance apparatus includes a state quantity acquisition means for acquiring a state quantity representing the amount of deflection or tension of the conveyance object in a plurality of sections of the conveyance path of the conveyance apparatus, and the conveyance apparatus sequentially conveys the conveyance target object by the plurality of conveyance means. sandwiching and transporting the object to be transported from upstream to downstream of the transport path,
Reward granting means for granting a reward based on the state quantity;
Learning means that performs machine learning to update an action value function that represents the value of a set of parameters of each driving means that drives each of the transport means, based on the reward;
determining one of the plurality of sets based on the updated action value function, and instructing the driving means to drive the conveying means with the parameters of the determined set; further comprising means;
The machine learning device communicates with a simulator that simulates the transport device,
The state quantity acquisition means acquires the state quantity based on the output from the simulator,
The reward granting means grants the reward based on the state quantity and the state of the transport means,
A predetermined transport means among the plurality of transport means transports the transport objects one by one from a storage means storing a plurality of transport objects to the transport path,
The reward giving means is configured such that the state quantity at a position before the rear end of the conveyance object reaches the predetermined conveyance means among the plurality of sections represents the pulling amount, and A machine learning device that provides a positive reward when the predetermined transport means is stopped when the rear end of the vehicle passes through the predetermined transport means . A machine learning device,
The conveyance apparatus includes a state quantity acquisition means for acquiring a state quantity representing the amount of deflection or tension of the conveyance object in a plurality of sections of the conveyance path of the conveyance apparatus, and the conveyance apparatus sequentially conveys the conveyance target object by the plurality of conveyance means. sandwiching and transporting the object to be transported from upstream to downstream of the transport path,
Reward granting means for granting a reward based on the state quantity;
Learning means that performs machine learning to update an action value function that represents the value of a set of parameters of each driving means that drives each of the transport means, based on the reward;
determining one of the plurality of sets based on the updated action value function, and instructing the driving means to drive the conveying means with the parameters of the determined set; further comprising means;
The machine learning device communicates with a simulator that simulates the transport device,
The state quantity acquisition means acquires the state quantity based on the output from the simulator,
The reward giving means gives the reward based on the state quantity and the state of the transport means,
The plurality of conveyance means includes a first conveyance means and a second conveyance means that is the next conveyance means downstream of the first conveyance means,
In the section between the first conveying means and the second conveying means among the plurality of sections, the object to be conveyed is deflected, and the object to be conveyed in the conveying direction in the second conveying means is If the generation of force is not permitted, the reward giving means determines whether the state quantity in the section between the first conveyance means and the second conveyance means is equal to the amount of tension or the amount of deflection. A machine learning device that gives a positive reward when the user does not represent the same . A machine learning device,
The conveyance apparatus includes a state quantity acquisition means for acquiring a state quantity representing the amount of deflection or tension of the conveyance object in a plurality of sections of the conveyance path of the conveyance apparatus, and the conveyance apparatus sequentially conveys the conveyance target object by the plurality of conveyance means. sandwiching and transporting the object to be transported from upstream to downstream of the transport path,
Reward granting means for granting a reward based on the state quantity;
Learning means that performs machine learning to update an action value function that represents the value of a set of parameters of each driving means that drives each of the transport means, based on the reward;
determining one of the plurality of sets based on the updated action value function, and instructing the driving means to drive the conveying means with the parameters of the determined set; further comprising means;
The machine learning device communicates with a simulator that simulates the transport device,
The state quantity acquisition means acquires the state quantity based on the output from the simulator,
The reward giving means gives the reward based on the state quantity and the state of the transport means,
The plurality of conveyance means includes a first conveyance means and a second conveyance means that is the next conveyance means downstream of the first conveyance means,
When it is possible to speed up the time for the object to be transported through the first transport means by transporting the object in a pulled state by the second transport means, the reward giving means , a machine learning device that gives a positive reward when the second conveying means conveys the conveyed object in a pulled state . A machine learning device,
The conveying device includes a state quantity acquisition means for acquiring a state quantity representing the amount of deflection or tension of the conveyed object in a plurality of sections of the conveyance path of the conveyance device, and the conveyance device sequentially conveys the conveyed object by the plurality of conveyance means. sandwiching and transporting the object to be transported from upstream to downstream of the transport path,
Reward granting means for granting a reward based on the state quantity;
Learning means that performs machine learning to update an action value function representing the value of a set of parameters of each driving means that drives each of the conveying means for each set based on the reward;
determining one of the plurality of sets based on the updated action value function, and instructing the driving means to drive the conveying means with the parameters of the determined set; further comprising means;
The machine learning device communicates with a simulator that simulates the transport device,
The state quantity acquisition means acquires the state quantity based on the output from the simulator,
The reward giving means gives the reward based on the state quantity and the state of the transport means,
The plurality of conveyance means includes a first conveyance means and a second conveyance means that is the next conveyance means downstream of the first conveyance means,
When the object to be transported is simultaneously transported by the first transport means and the second transport means, the area between the first transport means and the second transport means among the plurality of sections is When the setting is made to allow a predetermined amount of deflection in the section, the remuneration granting means provides, on the condition that the conveyance speed of the first conveyance means is equal to or higher than the conveyance speed of the second conveyance means, A machine learning device that gives positive rewards . A machine learning device,
The conveyance apparatus includes a state quantity acquisition means for acquiring a state quantity representing the amount of deflection or tension of the conveyance object in a plurality of sections of the conveyance path of the conveyance apparatus, and the conveyance apparatus sequentially conveys the conveyance target object by the plurality of conveyance means. sandwiching and transporting the object to be transported from upstream to downstream of the transport path,
Reward granting means for granting a reward based on the state quantity;
Learning means that performs machine learning to update an action value function that represents the value of a set of parameters of each driving means that drives each of the transport means, based on the reward;
determining one of the plurality of sets based on the updated action value function, and instructing the driving means to drive the conveying means with the parameters of the determined set; further comprising means;
The machine learning device communicates with a simulator that simulates the transport device,
The state quantity acquisition means acquires the state quantity based on the output from the simulator,
The reward giving means gives the reward based on the state quantity and the state of the transport means,
The plurality of conveyance means includes a first conveyance means and a second conveyance means that is the next conveyance means downstream of the first conveyance means,
When the first conveyance means and the second conveyance means are conveying the object at the same time, the distance between the first conveyance means and the second conveyance means among the plurality of sections is If a setting is made in which the deflection of the conveyed object is not allowed in the section, the remuneration granting means sets the condition that the conveying speed of the first conveying means is equal to or less than the conveying speed of the second conveying means. A machine learning device that gives positive rewards to people . The conveyance apparatus includes a state quantity acquisition means for acquiring a state quantity representing the amount of deflection or tension of the conveyance object in a plurality of sections of the conveyance path of the conveyance apparatus, and the conveyance apparatus sequentially conveys the conveyance target object by the plurality of conveyance means. sandwiching and transporting the object to be transported from upstream to downstream of the transport path,
Reward granting means for granting a reward based on the state quantity;
Learning means that performs machine learning to update an action value function that represents the value of a set of parameters of each driving means that drives each of the transport means, based on the reward;
determining one of the plurality of sets based on the updated action value function, and instructing the driving means to drive the conveying means with the parameters of the determined set; further comprising means;
The learning means performs machine learning to update the value of the parameter of each of the driving means based on the reward and the physical properties of the conveyed object,
The physical property is stiffness,
The plurality of conveyance means includes a first conveyance means and a second conveyance means that is the next conveyance means downstream of the first conveyance means,
When the first conveyance means and the second conveyance means are conveying the conveyed object at the same time, the remuneration giving means is configured such that the rigidity of the conveyed object is equal to or higher than a predetermined value and A machine learning device that provides a positive reward on the condition that the transport speed of the first transport means and the transport speed of the second transport means are the same . The conveyance apparatus includes a state quantity acquisition means for acquiring a state quantity representing the amount of deflection or tension of the conveyance object in a plurality of sections of the conveyance path of the conveyance apparatus, and the conveyance apparatus sequentially conveys the conveyance target object by the plurality of conveyance means. sandwiching and transporting the object to be transported from upstream to downstream of the transport path,
Reward granting means for granting a reward based on the state quantity;
Learning means that performs machine learning to update an action value function that represents the value of a set of parameters of each driving means that drives each of the transport means, based on the reward;
determining one of the plurality of sets based on the updated action value function, and instructing the driving means to drive the conveying means with the parameters of the determined set; further comprising means;
The learning means performs machine learning to update the value of the parameter of each of the driving means based on the reward and the physical properties of the conveyed object,
The physical property is stiffness,
The plurality of conveyance means includes a first conveyance means and a second conveyance means that is the next conveyance means downstream of the first conveyance means,
The first conveyance means and the second conveyance means convey the object to be conveyed at the same time, and the distance between the first conveyance means and the second conveyance means among the plurality of sections is In the section, when a setting is made to allow a predetermined amount of deflection, the reward giving means is configured to allow the stiffness of the conveyed object to be less than a predetermined value, and the first conveying means and the second conveying means A machine learning device that provides a positive reward when the state quantity in the interval between represents a deflection amount that is equal to or less than the predetermined deflection amount . The conveying device includes a state quantity acquisition means for acquiring a state quantity representing the amount of deflection or tension of the conveyed object in a plurality of sections of the conveyance path of the conveyance device, and the conveyance device sequentially conveys the conveyed object by the plurality of conveyance means. sandwiching and transporting the object to be transported from upstream to downstream of the transport path,
Reward granting means for granting a reward based on the state quantity;
Learning means that performs machine learning to update an action value function representing the value of a set of parameters of each driving means that drives each of the conveying means for each set based on the reward;
determining one of the plurality of sets based on the updated action value function, and instructing the driving means to drive the conveying means with the parameters of the determined set; further comprising means;
The learning means performs machine learning to update the value of the parameter of each of the driving means based on the reward and the physical properties of the conveyed object,
The physical property is basis weight,
The plurality of conveyance means includes a first conveyance means and a second conveyance means that is the next conveyance means downstream of the first conveyance means,
When the first conveying means and the second conveying means are simultaneously conveying the conveyed object, the remuneration granting means may be arranged such that the basis weight of the conveyed object is equal to or greater than a predetermined value, and the A machine learning device that provides a positive reward on the condition that the transport speed of the first transport means and the transport speed of the second transport means are the same . The conveyance apparatus includes a state quantity acquisition means for acquiring a state quantity representing the amount of deflection or tension of the conveyance object in a plurality of sections of the conveyance path of the conveyance apparatus, and the conveyance apparatus sequentially conveys the conveyance target object by the plurality of conveyance means. sandwiching and transporting the object to be transported from upstream to downstream of the transport path,
Reward granting means for granting a reward based on the state quantity;
Learning means that performs machine learning to update an action value function that represents the value of a set of parameters of each driving means that drives each of the transport means, based on the reward;
determining one of the plurality of sets based on the updated action value function, and instructing the driving means to drive the conveying means with the parameters of the determined set; further comprising means;
The learning means performs machine learning to update the value of the parameter of each of the driving means based on the reward and the physical properties of the conveyed object,
The physical property is basis weight,
The plurality of conveyance means includes a first conveyance means and a second conveyance means that is the next conveyance means downstream of the first conveyance means,
The first conveyance means and the second conveyance means convey the object to be conveyed at the same time, and the distance between the first conveyance means and the second conveyance means among the plurality of sections is When a setting is made to allow a predetermined amount of deflection in the section, the remuneration means may be configured to allow the first conveyance means and the second conveyance means when the basis weight of the conveyed object is less than a predetermined value. A machine learning device that provides a positive reward when the state amount in the interval between . The action value function is a Q table,
The machine learning device according to any one of claims 8 to 14 , wherein the determining means determines one set from a plurality of sets based on the acquired state quantity and the Q table. The state quantity acquisition means further acquires the state quantity when the conveyance means is driven based on the selected set of parameters,
The reward granting means further grants the reward based on the obtained state quantity,
The machine learning device according to any one of claims 1 to 19 , wherein the learning means further updates the action value function based on the given reward. The machine learning device according to any one of claims 1 to 20 , wherein the set includes at least one of speed, drive timing, stop timing, shift timing, and drive current value. The machine learning device according to any one of claims 1 to 21 , wherein the conveyed object is a sheet of paper. The machine learning device according to any one of claims 1 to 21 , wherein the conveyed object is cloth. The machine learning device according to any one of claims 1 to 23 , wherein the learning means performs machine learning to update the value of the parameter of each of the driving means based on the reward and the physical properties of the conveyed object. . The machine learning device according to claim 24 , wherein the physical property is stiffness. The machine learning device according to claim 24 , wherein the physical property is basis weight. The machine learning device according to any one of claims 1 to 26 , wherein the conveyance means is a pair of rollers. The machine learning device according to claim 1 , wherein the predetermined amount of deflection is a value less than the width of the conveyance path in the thickness direction of the conveyed object. The machine learning device according to claim 1 or 8 , wherein the predetermined amount of deflection is a predetermined value larger than zero. The machine learning device according to any one of claims 1 to 29 , wherein the state quantity acquisition means acquires the deflection amount using a simulation model of a mechanical sensor provided in the transport device. The machine learning device according to any one of claims 1 to 29 , wherein the state quantity acquisition means acquires the deflection amount using a simulation model of an optical sensor provided in the transport device. The state quantity acquisition means is
obtaining the length of the object to be transported based on the position of the object to be transported;
According to any one of claims 1 to 29 , when the reference length of the conveyed object is longer than the obtained length, the difference between the obtained length and the reference length is set as the amount of deflection. Machine learning device described. The state quantity acquisition means is
Obtaining the load of the conveyance means using a simulation model of a load detection means provided in the conveyance apparatus,
The machine learning device according to any one of claims 1 to 29 , wherein the amount of tension is obtained based on the load. The machine learning device according to any one of claims 1 to 29 , wherein the state quantity acquisition means acquires the amount of tension using a simulation model of an optical sensor provided in the transport device. The plurality of conveyance means includes a first conveyance means and a second conveyance means that is the next conveyance means downstream of the first conveyance means,
The state quantity acquisition means is
obtaining the length of the object to be transported based on the position of the object to be transported;
If there is no difference between the acquired length and the reference length of the object to be transported, and the transport speed of the second transport means is equal to or higher than the transport speed of the first transport means, the object to be transported is pulled. The machine learning device according to any one of claims 1 to 29 , wherein the machine learning device determines that the state is the same. According to any one of claims 1 to 35 , the machine learning is executed again when the conveyance speed of the conveyance object of the conveyance means is different from the conveyance speed set during the previous machine learning. Machine learning device described. The machine learning device according to any one of claims 1 to 36 , wherein a control program for updating including the parameters as a result of the machine learning is transmitted to a controller that controls operation of the transport device. A conveyance device comprising the machine learning device according to any one of claims 1 to 37 . An image forming apparatus comprising the machine learning device according to claim 1 . a step of acquiring state quantities representing the amount of deflection or tension of the conveyed object in a plurality of sections of a conveying path of a conveying device, the conveying device sequentially holding the conveyed object by a plurality of conveying means; , transporting the object to be transported from upstream to downstream of the transport path,
a step of providing a reward based on the state quantity;
updating, based on the reward, an action value function that represents the value of a set of parameters of each drive means that drives each of the transport means for each set;
determining one of the plurality of sets based on the updated action value function, and instructing the driving means to drive the conveying means with the parameters of the determined set; and,
further comprising the step of acquiring the state quantity based on the conveyance speed of the conveyance target by the conveyance means or the position of the conveyance target in the conveyance path,
The plurality of conveyance means includes a first conveyance means and a second conveyance means that is the next conveyance means downstream of the first conveyance means,
If a setting is made to allow a predetermined amount of deflection in the section between the first conveying means and the second conveying means among the plurality of sections, the first conveying means and the second conveying means Machine learning, wherein when the state quantity in the section between the transport means and the conveyance means represents the deflection amount that is less than or equal to the predetermined deflection amount, the step of giving a reward based on the state quantity gives a positive reward . Method. a step of acquiring state quantities representing the amount of deflection or tension of the conveyed object in a plurality of sections of a conveying path of a conveying device, the conveying device sequentially holding the conveyed object by a plurality of conveying means; , transporting the object to be transported from upstream to downstream of the transport path,
a step of providing a reward based on the state quantity;
updating, based on the reward, an action value function that represents the value of a set of parameters of each drive means that drives each of the transport means for each set;
determining one of the plurality of sets based on the updated action value function, and instructing the driving means to drive the conveying means with the parameters of the determined set; and,
further comprising the step of acquiring the state quantity based on the conveyance speed of the conveyance target by the conveyance means or the position of the conveyance target in the conveyance path,
The plurality of conveyance means includes a first conveyance means and a second conveyance means that is the next conveyance means downstream of the first conveyance means,
In a section between the first conveying means and the second conveying means among the plurality of sections, if a setting is made that does not allow deflection of the conveyed object, the first conveying means and the second conveying means A machine learning method , wherein when the state quantity in the section between the second transport means and the second transport means represents the amount of pull, a positive reward is given in the step of giving a reward based on the state quantity . a step of acquiring state quantities representing the amount of deflection or tension of the conveyed object in a plurality of sections of a conveying path of a conveying device, the conveying device sequentially holding the conveyed object by a plurality of conveying means; , transporting the object to be transported from upstream to downstream of the transport path,
a step of granting a reward based on the state quantity;
updating, based on the reward, an action value function that represents the value of a set of parameters of each drive means that drives each of the transport means for each set;
determining one of the plurality of sets based on the updated action value function, and instructing the driving means to drive the conveying means with the parameters of the determined set; and,
further comprising the step of acquiring the state quantity based on the conveyance speed of the conveyance target by the conveyance means or the position of the conveyance target in the conveyance path,
In the step of providing a reward based on the state quantity, the reward is provided based on the state quantity and the state of the transport means,
A predetermined transport means among the plurality of transport means transports the transport objects one by one from a storage means storing a plurality of transport objects to the transport path,
Among the plurality of sections, the state quantity at a position before the rear end of the conveyance target reaches the predetermined conveyance means represents the amount of tension, and the rear end of the conveyance target reaches the predetermined conveyance means. If the predetermined transport means is stopped when passing through the transport means, a positive reward is given in the step of giving a reward based on the state quantity . a step of acquiring state quantities representing the amount of deflection or tension of the conveyed object in a plurality of sections of a conveying path of a conveying device, the conveying device sequentially holding the conveyed object by a plurality of conveying means; , transporting the object to be transported from upstream to downstream of the transport path,
a step of providing a reward based on the state quantity;
updating, based on the reward, an action value function that represents the value of a set of parameters of each drive means that drives each of the transport means for each set;
determining one of the plurality of sets based on the updated action value function, and instructing the driving means to drive the conveying means with the parameters of the determined set; and,
further comprising the step of acquiring the state quantity based on the conveyance speed of the conveyance target by the conveyance means or the position of the conveyance target in the conveyance path,
In the step of providing the reward based on the state quantity, the reward is provided based on the state quantity and the state of the conveying means,
The plurality of conveyance means includes a first conveyance means and a second conveyance means that is the next conveyance means downstream of the first conveyance means,
In the section between the first conveying means and the second conveying means among the plurality of sections, the object to be conveyed is deflected, and the object to be conveyed in the conveying direction in the second conveying means is If the generation of force is not allowed, and the state quantity in the section between the first conveying means and the second conveying means does not represent either the amount of tension or the amount of deflection, A machine learning method , wherein a positive reward is given in the step of giving a reward based on the state amount . a step of acquiring state quantities representing the amount of deflection or tension of the conveyed object in a plurality of sections of a conveying path of a conveying device, the conveying device sequentially holding the conveyed object by a plurality of conveying means; , transporting the object to be transported from upstream to downstream of the transport path,
a step of providing a reward based on the state quantity;
updating, based on the reward, an action value function that represents the value of a set of parameters of each drive means that drives each of the transport means for each set;
determining one of the plurality of sets based on the updated action value function, and instructing the driving means to drive the conveying means with the parameters of the determined set; and,
further comprising the step of acquiring the state quantity based on the conveyance speed of the conveyance target by the conveyance means or the position of the conveyance target in the conveyance path,
In the step of providing a reward based on the state quantity, the reward is provided based on the state quantity and the state of the transport means,
The plurality of conveyance means includes a first conveyance means and a second conveyance means that is the next conveyance means downstream of the first conveyance means,
If it is possible to speed up the time for the object to be transported through the first transport means by transporting the object in a pulled state by the second transport means, the second transport means A machine learning method , wherein when the object is being transported in a pulled state, a positive reward is provided in the step of providing a reward based on the state quantity . a step of acquiring state quantities representing the amount of deflection or tension of the conveyed object in a plurality of sections of a conveying path of a conveying device, the conveying device sequentially holding the conveyed object by a plurality of conveying means; , transporting the object to be transported from upstream to downstream of the transport path,
a step of granting a reward based on the state quantity;
updating, based on the reward, an action value function that represents the value of a set of parameters of each drive means that drives each of the transport means for each set;
determining one of the plurality of sets based on the updated action value function, and instructing the driving means to drive the conveying means with the parameters of the determined set; and,
further comprising the step of acquiring the state quantity based on the conveyance speed of the conveyance target by the conveyance means or the position of the conveyance target in the conveyance path,
In the step of providing a reward based on the state quantity, the reward is provided based on the state quantity and the state of the transport means,
The plurality of conveyance means includes a first conveyance means and a second conveyance means that is the next conveyance means downstream of the first conveyance means,
When the first conveyance means and the second conveyance means are conveying the object at the same time, the distance between the first conveyance means and the second conveyance means among the plurality of sections is When a setting is made to allow a predetermined amount of deflection in the section, in the step of giving a reward based on the state quantity, the conveyance speed of the first conveyance means is equal to or higher than the conveyance speed of the second conveyance means. A machine learning method that gives positive rewards on the condition that . a step of acquiring state quantities representing the amount of deflection or tension of the conveyed object in a plurality of sections of a conveying path of a conveying device, the conveying device sequentially holding the conveyed object by a plurality of conveying means; , transporting the object to be transported from upstream to downstream of the transport path,
a step of providing a reward based on the state quantity;
updating, based on the reward, an action value function that represents the value of a set of parameters of each drive means that drives each of the transport means for each set;
determining one of the plurality of sets based on the updated action value function, and instructing the driving means to drive the conveying means with the parameters of the determined set; and,
further comprising the step of acquiring the state quantity based on the conveyance speed of the conveyance target by the conveyance means or the position of the conveyance target in the conveyance path,
In the step of providing a reward based on the state quantity, the reward is provided based on the state quantity and the state of the transport means,
The plurality of conveyance means includes a first conveyance means and a second conveyance means that is the next conveyance means downstream of the first conveyance means,
When the first conveyance means and the second conveyance means are conveying the object at the same time, the distance between the first conveyance means and the second conveyance means among the plurality of sections is If the setting is such that the deflection of the conveyed object is not allowed in the section, in the step of giving a reward based on the state quantity, the conveying speed of the first conveying means is set to be lower than the conveying speed of the second conveying means. A machine learning method that gives a positive reward on the condition that the speed is lower than or equal to the speed . a step of acquiring state quantities representing the amount of deflection or tension of the conveyed object in a plurality of sections of a conveying path of a conveying device, the conveying device sequentially holding the conveyed object by a plurality of conveying means; , transporting the object to be transported from upstream to downstream of the transport path,
a step of granting a reward based on the state quantity;
updating, based on the reward, an action value function that represents the value of a set of parameters of each drive means that drives each of the transport means for each set;
determining one of the plurality of sets based on the updated action value function, and instructing the driving means to drive the conveying means with the parameters of the determined set; and,
communicating with a simulator simulating the transport device;
further comprising the step of acquiring the state quantity based on the output from the simulator,
The plurality of conveyance means includes a first conveyance means and a second conveyance means that is the next conveyance means downstream of the first conveyance means,
If a setting is made to allow a predetermined amount of deflection in the section between the first conveying means and the second conveying means among the plurality of sections, the first conveying means and the second conveying means The step of awarding a reward based on the state quantity provides a positive reward when the state quantity represents the deflection amount that is less than or equal to the predetermined deflection amount in the section between the machine and the conveying means. How to learn. a step of acquiring state quantities representing the amount of deflection or tension of the conveyed object in a plurality of sections of a conveying path of a conveying device, the conveying device sequentially holding the conveyed object by a plurality of conveying means; , transporting the object to be transported from upstream to downstream of the transport path,
a step of granting a reward based on the state quantity;
updating, based on the reward, an action value function that represents the value of a set of parameters of each drive means that drives each of the transport means for each set;
determining one of the plurality of sets based on the updated action value function, and instructing the driving means to drive the conveying means with the parameters of the determined set; and,
communicating with a simulator simulating the transport device;
further comprising the step of acquiring the state quantity based on the output from the simulator,
The plurality of conveyance means includes a first conveyance means and a second conveyance means that is the next conveyance means downstream of the first conveyance means,
In a section between the first conveying means and the second conveying means among the plurality of sections, if a setting is made that does not allow deflection of the conveyed object, the first conveying means and the second conveying means A machine learning method , wherein when the state quantity in the section between the second transport means and the second transport means represents the amount of tension, a positive reward is given in the step of giving a reward based on the state quantity. a step of acquiring state quantities representing the amount of deflection or tension of the conveyed object in a plurality of sections of a conveying path of a conveying device, the conveying device sequentially holding the conveyed object by a plurality of conveying means; , transporting the object to be transported from upstream to downstream of the transport path,
a step of providing a reward based on the state quantity;
updating, based on the reward, an action value function that represents the value of a set of parameters of each drive means that drives each of the transport means for each set;
determining one of the plurality of sets based on the updated action value function, and instructing the driving means to drive the conveying means with the parameters of the determined set; and,
communicating with a simulator simulating the transport device;
further comprising the step of obtaining the state quantity based on the output from the simulator,
In the step of providing a reward based on the state quantity, the reward is provided based on the state quantity and the state of the transport means,
A predetermined transport means among the plurality of transport means transports the transport objects one by one from a storage means storing a plurality of transport objects to the transport path,
Among the plurality of sections, the state quantity at a position before the rear end of the conveyance target reaches the predetermined conveyance means represents the amount of tension, and the rear end of the conveyance target reaches the predetermined conveyance means. If the predetermined transport means is stopped when passing through the transport means, a positive reward is given in the step of giving a reward based on the state quantity . a step of acquiring state quantities representing the amount of deflection or tension of the conveyed object in a plurality of sections of a conveying path of a conveying device, the conveying device sequentially holding the conveyed object by a plurality of conveying means; , transporting the object to be transported from upstream to downstream of the transport path,
a step of providing a reward based on the state quantity;
updating, based on the reward, an action value function that represents the value of a set of parameters of each drive means that drives each of the transport means for each set;
determining one of the plurality of sets based on the updated action value function, and instructing the driving means to drive the conveying means with the parameters of the determined set; and,
communicating with a simulator simulating the transport device;
further comprising the step of obtaining the state quantity based on the output from the simulator,
In the step of providing the reward based on the state quantity, the reward is provided based on the state quantity and the state of the conveying means,
The plurality of conveyance means includes a first conveyance means and a second conveyance means that is the next conveyance means downstream of the first conveyance means,
In the section between the first conveying means and the second conveying means among the plurality of sections, the object to be conveyed is deflected, and the object to be conveyed in the conveying direction in the second conveying means is If the generation of force is not allowed, and the state quantity in the section between the first conveying means and the second conveying means does not represent either the amount of tension or the amount of deflection, A machine learning method , wherein a positive reward is given in the step of giving a reward based on the state amount . a step of acquiring state quantities representing the amount of deflection or tension of the conveyed object in a plurality of sections of a conveying path of a conveying device, the conveying device sequentially holding the conveyed object by a plurality of conveying means; , transporting the object to be transported from upstream to downstream of the transport path,
a step of providing a reward based on the state quantity;
updating, based on the reward, an action value function that represents the value of a set of parameters of each drive means that drives each of the transport means for each set;
determining one of the plurality of sets based on the updated action value function, and instructing the driving means to drive the conveying means with the parameters of the determined set; and,
communicating with a simulator simulating the transport device;
further comprising the step of obtaining the state quantity based on the output from the simulator,
In the step of providing the reward based on the state quantity, the reward is provided based on the state quantity and the state of the conveying means,
The plurality of conveyance means includes a first conveyance means and a second conveyance means that is the next conveyance means downstream of the first conveyance means,
When it is possible to speed up the time for the object to be transported through the first transport means by transporting the object in a pulled state by the second transport means, the second transport means A machine learning method , in which a positive reward is provided in the step of providing a reward based on the state quantity when the object is being transported in a pulled state . a step of acquiring state quantities representing the amount of deflection or tension of the conveyed object in a plurality of sections of a conveying path of a conveying device, the conveying device sequentially holding the conveyed object by a plurality of conveying means; , transporting the object to be transported from upstream to downstream of the transport path,
a step of providing a reward based on the state quantity;
updating, based on the reward, an action value function that represents the value of a set of parameters of each drive means that drives each of the transport means for each set;
determining one of the plurality of sets based on the updated action value function, and instructing the driving means to drive the conveying means with the parameters of the determined set; and,
communicating with a simulator simulating the transport device;
further comprising the step of obtaining the state quantity based on the output from the simulator,
In the step of providing the reward based on the state quantity, the reward is provided based on the state quantity and the state of the conveying means,
The plurality of conveyance means includes a first conveyance means and a second conveyance means that is the next conveyance means downstream of the first conveyance means,
When the object to be transported is simultaneously transported by the first transport means and the second transport means, the area between the first transport means and the second transport means among the plurality of sections is When a setting is made to allow a predetermined amount of deflection in the section, in the step of giving a reward based on the state quantity, the conveyance speed of the first conveyance means is equal to or higher than the conveyance speed of the second conveyance means. A machine learning method that gives positive rewards on the condition that . a step of acquiring state quantities representing the amount of deflection or tension of the conveyed object in a plurality of sections of a conveying path of a conveying device, the conveying device sequentially holding the conveyed object by a plurality of conveying means; , transporting the object to be transported from upstream to downstream of the transport path,
a step of providing a reward based on the state quantity;
updating, based on the reward, an action value function that represents the value of a set of parameters of each drive means that drives each of the transport means for each set;
determining one of the plurality of sets based on the updated action value function, and instructing the driving means to drive the conveying means with the parameters of the determined set; and,
communicating with a simulator simulating the transport device;
further comprising the step of obtaining the state quantity based on the output from the simulator,
In the step of providing a reward based on the state quantity, the reward is provided based on the state quantity and the state of the transport means,
The plurality of conveyance means includes a first conveyance means and a second conveyance means that is the next conveyance means downstream of the first conveyance means,
When the first conveyance means and the second conveyance means are conveying the object at the same time, the distance between the first conveyance means and the second conveyance means among the plurality of sections is If the setting is such that the deflection of the conveyed object is not allowed in the section, in the step of giving a reward based on the state quantity, the conveying speed of the first conveying means is set to be lower than the conveying speed of the second conveying means. A machine learning method that gives a positive reward on the condition that the speed is lower than or equal to the speed . a step of acquiring state quantities representing the amount of deflection or tension of the conveyed object in a plurality of sections of a conveying path of a conveying device, the conveying device sequentially holding the conveyed object by a plurality of conveying means; , transporting the object to be transported from upstream to downstream of the transport path,
a step of granting a reward based on the state quantity;
updating, based on the reward, an action value function that represents the value of a set of parameters of each drive means that drives each of the transport means for each set;
determining one of the plurality of sets based on the updated action value function, and instructing the driving means to drive the conveying means with the parameters of the determined set; and,
further comprising the step of performing machine learning for updating parameter values of each of the driving means based on the reward and the physical properties of the conveyed object,
The physical property is stiffness,
The plurality of conveyance means includes a first conveyance means and a second conveyance means that is the next conveyance means downstream of the first conveyance means,
When the first conveying means and the second conveying means are simultaneously conveying the conveyed object, in the step of providing a reward based on the state quantity, the stiffness of the conveyed object is equal to or higher than a predetermined value. A machine learning method that provides a positive reward on the condition that the transport speed of the first transport means and the transport speed of the second transport means are the same . a step of acquiring state quantities representing the amount of deflection or tension of the conveyed object in a plurality of sections of a conveying path of a conveying device, the conveying device sequentially holding the conveyed object by a plurality of conveying means; , transporting the object to be transported from upstream to downstream of the transport path,
a step of providing a reward based on the state quantity;
updating, based on the reward, an action value function that represents the value of a set of parameters of each drive means that drives each of the transport means for each set;
determining one of the plurality of sets based on the updated action value function, and instructing the driving means to drive the conveying means with the parameters of the determined set; and,
further comprising the step of performing machine learning for updating parameter values of each of the driving means based on the reward and the physical properties of the conveyed object,
The physical property is stiffness,
The plurality of conveyance means includes a first conveyance means and a second conveyance means that is the next conveyance means downstream of the first conveyance means,
The first conveyance means and the second conveyance means convey the object to be conveyed at the same time, and the distance between the first conveyance means and the second conveyance means among the plurality of sections is If a setting is made to allow a predetermined amount of deflection in the section, the stiffness of the object to be transported is less than a predetermined value, and the stiffness of the object in the section between the first transport means and the second transport means is A machine learning method , wherein when the amount of state represents an amount of deflection that is less than or equal to the predetermined amount of deflection, a positive reward is provided in the step of providing a reward based on the amount of state . a step of acquiring state quantities representing the amount of deflection or tension of the conveyed object in a plurality of sections of a conveying path of a conveying device, the conveying device sequentially holding the conveyed object by a plurality of conveying means; , transporting the object to be transported from upstream to downstream of the transport path,
a step of granting a reward based on the state quantity;
updating, based on the reward, an action value function that represents the value of a set of parameters of each drive means that drives each of the transport means for each set;
determining one of the plurality of sets based on the updated action value function, and instructing the driving means to drive the conveying means with the parameters of the determined set; and,
further comprising the step of performing machine learning for updating parameter values of each of the driving means based on the reward and the physical properties of the conveyed object,
The physical property is basis weight,
The plurality of conveyance means includes a first conveyance means and a second conveyance means that is the next conveyance means downstream of the first conveyance means,
When the object to be transported is simultaneously transported by the first transport means and the second transport means, in the step of providing a reward based on the state quantity, the basis weight of the object to be transported is a predetermined value. A machine learning method that provides a positive reward on the condition that the above is the case, and that the transport speed of the first transport means and the transport speed of the second transport means are the same . a step of acquiring state quantities representing the amount of deflection or tension of the conveyed object in a plurality of sections of a conveying path of a conveying device, the conveying device sequentially holding the conveyed object by a plurality of conveying means; , transporting the object to be transported from upstream to downstream of the transport path,
a step of providing a reward based on the state quantity;
updating, based on the reward, an action value function that represents the value of a set of parameters of each drive means that drives each of the transport means for each set;
determining one of the plurality of sets based on the updated action value function, and instructing the driving means to drive the conveying means with the parameters of the determined set; and,
further comprising the step of performing machine learning for updating parameter values of each of the driving means based on the reward and the physical properties of the conveyed object,
The physical property is basis weight,
The plurality of conveyance means includes a first conveyance means and a second conveyance means that is the next conveyance means downstream of the first conveyance means,
The first conveyance means and the second conveyance means convey the object to be conveyed at the same time, and the distance between the first conveyance means and the second conveyance means among the plurality of sections is If a setting is made to allow a predetermined amount of deflection in the section, the basis weight of the object to be transported is less than a predetermined value, and in the section between the first transport means and the second transport means. A machine learning method , wherein when the amount of state represents a deflection amount that is less than or equal to the predetermined amount of deflection, a positive reward is provided in the step of providing a reward based on the amount of state . A program that causes a processor of a computer to execute each step according to any one of claims 40 to 57 .

Images