JP6725930B1 - Liquid weighing method, control device, computer program and learning method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a liquid weighing method, a controller, a computer program, and a learning method for weighing water by pouring water from a first container to a second container using a robot arm. A liquid weighing method according to the present embodiment receives an image of a first container, an angle of the first container, and a liquid amount of a second container at a first time point as inputs, and the liquid weighing method at a second time point. A controller for controlling the operation of the robot arm using a learning model machine-learned to output the angle of the first container acquires an image including the first container captured at a first time point, and The angle of the first container at one time point is acquired, the liquid amount of the second container at the first time point is acquired, and the acquired image, angle, and liquid amount are input to the learning model, and the learning model is The angle at the second time point to be output is acquired, and the angle of the first container held by the robot arm is controlled according to the acquired angle at the second time point. [Selection diagram] Fig. 5

Description

The present invention relates to a liquid weighing method, a control device, a computer program, and a learning method for weighing a liquid using a robot arm.

Conventionally, a multi-joint robot arm (manipulator) configured by connecting a plurality of joint units is known. The robot arm has high versatility, and is expected to be used in various industries such as assembling and carrying work in factories and remote work in laboratories.

Patent Document 1 proposes a sample pretreatment method that performs pretreatment when a plurality of types of tests are performed on a sample of the same subject. According to this method, the dispensing unit dispenses the sample contained in the first sample container into the plurality of second sample containers, and the robot arm holds the second sample container and conveys it to the sample pretreatment unit. As a result, the sample pretreatment method described in Patent Document 1 can reduce the work load on the user and can suppress the occurrence of contamination (sample contamination) due to the mixing of components originating from the user.

JP, 2019-174369, A

In the sample pretreatment method described in Patent Document 1, the robot arm is only used for transporting the second sample container, and the dispensing of the sample from the first sample container to the second sample container is performed by another device. It is being appreciated.

The present invention has been made in view of such circumstances, and an object thereof is to use a robot arm to inject water from a first container to a second container and measure a liquid. A liquid weighing method, a control device, a computer program, and a learning method are provided.

A liquid weighing method according to an embodiment is a liquid weighing method of pouring a predetermined amount of liquid from a first container held by a robot arm into a second container, and is an image of the first container at a first time point, A controller for controlling the operation of the robot arm by using a learning model machine-learned to receive the angle of one container and the liquid amount of the second container as inputs and output the angle of the first container at a second time point. Acquires an image including the first container taken at a first time point, acquires an angle of the first container at the first time point, and acquires a liquid amount of the second container at the first time point. The acquired image, the angle, and the liquid amount are input to the learning model to acquire the angle at the second time point output by the learning model, and the robot arm holds the angle according to the acquired angle at the second time point. The angle of the first container to be controlled is controlled.

According to one embodiment, it can be expected that water is weighed from the first container to the second container by using the robot arm.

It is a schematic diagram for explaining the outline of the liquid weighing system according to the present embodiment. It is a schematic diagram which shows one structural example of a robot arm. It is a block diagram which shows the structure of the control apparatus which concerns on this Embodiment. It is a schematic diagram for explaining the image acquisition process of the first container performed by the control device according to the present embodiment. It is a schematic diagram for demonstrating the structure of the learning model which concerns on this Embodiment. It is a schematic diagram for demonstrating the learning process of the encoder and decoder of CAE. It is a schematic diagram for demonstrating the learning process of a learning model. It is a schematic diagram for demonstrating the limitation of a sample operation. It is a schematic diagram for demonstrating the limitation of a sample operation. 5 is a flowchart showing a procedure of teacher data generation processing performed by the control device according to the present embodiment. 5 is a flowchart showing a procedure of learning processing of a learning model performed by the control device according to the present embodiment. It is a flow chart which shows a schematic procedure of liquid weighing processing which a control device concerning this embodiment performs. It is a flow chart which shows a detailed procedure of liquid weighing processing which a control device concerning this embodiment performs.

A specific example of the liquid weighing system according to the embodiment of the present invention will be described below with reference to the drawings. It should be noted that the present invention is not limited to these exemplifications, and is shown by the scope of the claims, and is intended to include meanings equivalent to the scope of the claims and all modifications within the scope.

<System overview>
FIG. 1 is a schematic diagram for explaining the outline of the liquid weighing system according to the present embodiment. The liquid weighing system according to the present embodiment is configured to include a control device 1, a robot arm 2, a camera 3, a measuring device 4, and the like. In the liquid weighing system according to the present embodiment, the robot arm 2 holds the first container 101 containing the liquid and conveys the first container 101 from the container storage area 104 of the work table 103 to the mounting area of the second container 102. 2 tilts the first container 101 to inject water from the first container 101 into the second container 102. The control device 1 controls the operation of the robot arm 2 so as to inject a predetermined amount of liquid from the first container 101 into the second container 102, and thereby weighs a predetermined amount of liquid.

The liquid to be weighed by the liquid weighing system according to this embodiment may be any liquid. The first container 101 and the second container 102 may have any shape and size. The first container 101 can be grasped by the robot arm 2 and tilted, and any liquid can be poured into the second container 102 by tilting it. The second container 102 can contain the liquid poured from the first container 101, and may be any container that can contain a predetermined amount of liquid. The robot arm 2 may be any one that can perform at least the operation of tilting the first container 101.

The camera 3 photographs the work of weighing performed on the workbench 103, and transmits time-series images (moving images) obtained by the photographing to the control device 1. In the present embodiment, the camera 3 images at least the first container 101 while the robot arm 2 is injecting water from the first container 101 into the second container 102. The image captured by the camera 3 at this time preferably includes the liquid surface of the liquid contained in the first container 101.

The measuring instrument 4 measures the amount (mass or volume) of the liquid poured into the second container 102, and sends the measurement result to the control device 1. In the liquid weighing system according to the present embodiment, the robot arm 2 injects the liquid from the first container 101 to the second container 102 so that the liquid amount measured by the measuring device 4 becomes a predetermined amount. .. In the present embodiment, the amount of liquid in the second container 102 is measured by the measuring device 4, but the present invention is not limited to this, and the amount of liquid in the second container 102 may be measured by any device. Good. For example, the liquid amount may be measured based on the image of the second container 102 taken by the camera 3.

The control device 1 is connected to the robot arm 2 and controls the operation of the robot arm 2 by controlling a motor, an actuator, or the like included in the robot arm 2. In the liquid weighing system according to the present embodiment, the control device 1 controls the inclination (angle) of the first container 101 held by the robot arm 2 to perform the weighing operation by the robot arm 2. The control device 1 does not need to be arranged in the vicinity of the robot arm 2, and may be a device that remotely controls the robot arm 2, such as a cloud server device.

Further, in the present embodiment, the image captured by the camera 3 and the liquid amount of the second container 102 measured by the measuring device 4 are input to the control device 1. Although detailed description will be given later, the control device 1 includes a learned learning model that has been machine-learned in advance, and uses this learning model to perform processing for controlling the operation of the robot arm 2. The learning model is preliminarily configured to output the angle of the first container 101 at the next time point with respect to the input of the angle of the first container 101 at the present time, the image of the first container 101, and the liquid amount of the second container 102. Machine learning is done. The control device 1 inputs the image of the first container 101 taken by the camera 3, the angle of the first container 101, and the liquid amount of the second container 102 weighed by the weighing machine 4 to the learning model, and the learning model The angle of the first container 101 at the next time point output by is acquired. The control device 1 controls the robot arm 2 by using the acquired angle at the next time as a target value, and changes the angle of the first container 101.

After completing the injection of the predetermined amount of liquid into the second container 102, the control device 1 controls the operation of the robot arm 2 so as to place the first container 101 on the container storage area 104. At this time, the container storage place 104 where the robot arm 2 mounts the first container 101 may be the same as the place where the first container 101 was initially placed, or may be a different place.

<Device configuration>
FIG. 2 is a schematic diagram showing a configuration example of the robot arm 2. The robot arm 2 according to this embodiment includes a first base portion 20a, a second base portion 20b, a first arm portion 20c, a second arm portion 20d, and a third arm portion 20e. The first base portion 20a constitutes a base of the robot arm 2, and is fixedly fixed to, for example, the work table 103 or a floor surface near the work table 103. The second base portion 20b is rotatably supported by the first base portion 20a via the first shaft 21.

The first arm portion 20c is a gently curved long plate-shaped member, and is swingably supported by the second base portion 20b via the second shaft 22 at one end side in the longitudinal direction. The other end of the first arm portion 20c supports the second arm portion 20d swingably via the third shaft 23. The second arm portion 20d is a long rod-shaped member, and the fourth shaft 24 is embedded in the middle portion thereof. A tip portion of the second arm portion 20d is rotatable about the axis by the fourth shaft 24 with respect to a root portion supported by the first arm portion 20c. The third arm portion 20e is swingably supported via the fifth shaft 25 at the tip of the second arm portion 20d. The third arm portion 20e is a short rod-shaped member, and the sixth shaft 26 is embedded in the middle portion thereof. The tip portion of the third arm portion 20e is rotatable by the sixth shaft 26 with respect to the root portion supported by the second arm portion 20d.

Various end effectors can be attached to the third arm portion 20e. As the end effector, various things such as a hand (gripper), a suction hand, a spoon, a spray gun or a welding torch can be attached. In the present embodiment, an end effector such as a hand capable of holding the first container 101 is attached to the third arm portion 20e of the robot arm 2. The first container 101 held by the end effector of the robot arm 2 is tilted by the rotation of the sixth shaft 26. That is, the control device 1 can control the angle of the first container 101 held by the end effector by controlling the rotation angle of the sixth shaft 26 of the robot arm 2.

The first axis 21 to the sixth axis 26 of the robot arm 2 are rotated by individual motors. The control device 1 can individually control the rotations of these motors to carry the first container 101 by the robot arm 2 and to inject liquid from the first container 101 to the second container 102. Further, each motor that rotates the first shaft 21 to the sixth shaft 26 of the robot arm 2 is a motor that can control the rotation angle with high accuracy, such as a stepping motor. The control device 1 can know the rotation angle of each motor and can control with high accuracy.

The robot arm 2 used in the liquid weighing system is not limited to that shown in FIG. The number of rotation axes of the robot arm 2 may be five or less, or seven or more, as long as it has at least one rotation axis for controlling the angle of the first container 101 gripped by the end effector. Good. Further, the number of base portions of the robot arm 2, the number of arm portions, the connecting structure of each member, and the like are not limited to those shown in the drawings, and may be appropriately changed.

FIG. 3 is a block diagram showing the configuration of the control device 1 according to the present embodiment. The control device 1 according to the present embodiment includes a processing unit 11, a storage unit (storage) 12, a communication unit (transceiver) 13, a display unit (display) 14, an operation unit 15, and the like. The processing unit 11 uses an arithmetic processing unit such as a CPU (Central Processing Unit), an MPU (Micro-Processing Unit) or a GPU (Graphics Processing Unit), a ROM (Read Only Memory), and a RAM (Random Access Memory). Is configured. The processing unit 11 reads and executes the control program 12a stored in the storage unit 12 to perform various processes such as a process for controlling the operation of the robot arm 2 and a learning model learning process.

The storage unit 12 is configured by using a large-capacity storage device such as a hard disk. The storage unit 12 stores various programs executed by the processing unit 11 and various data necessary for the processing of the processing unit 11. In the present embodiment, the storage unit 12 stores the control program 12a executed by the processing unit 11 and the learning model 5 that has been learned in advance.

In the present embodiment, the control program 12a is provided in a form recorded on a recording medium 99 such as a memory card or an optical disk, and the control device 1 reads the control program 12a from the recording medium 99 and stores it in the storage unit 12. However, the control program 12a may be written in the storage unit 12 at the manufacturing stage of the control device 1, for example. Further, for example, the control program 12a may be acquired by the control device 1 by communication through a program distributed by a remote server device. For example, the control program 12a may be recorded in the recording medium 99 by a writing device and written in the storage unit 12 of the control device 1. The control program 12a may be provided in the form of distribution via a network, or may be provided in the form recorded in the recording medium 99.

The learning model 5 included in the control device 1 according to the present embodiment is a learning model in which machine learning is performed so as to control the operation of the robot arm 2 and weigh the liquid. In the present embodiment, the control device 1 performs the learning process of the learning model 5 in advance, but the present invention is not limited to this. The learning model 5 may be preliminarily machine-learned by another server device or the like, and the control device 1 may pass the learned learning model preliminarily machine-learned by the server device or the like through the network or the recording medium 99. May be acquired.

The communication unit 13 transmits/receives data to/from devices such as the robot arm 2, the camera 3, and the weighing instrument 4 via a wired or wireless network. In the present embodiment, the control device 1 transmits, to the robot arm 2, a control command or the like for controlling the rotation of the first axis 21 to the sixth axis 26 to the robot arm 2 from the communication unit 13. Further, the control device 1 may receive various information fed back from the robot arm 2 at the communication unit 13. Further, the control device 1 receives data of time-series images (moving images) captured by the camera 3 from the camera 3 via the communication unit 13. Further, the control device 1 receives the data such as the mass or the capacity measured by the measuring instrument 4 from the measuring instrument 4 by the communication unit 13. The communication unit 13 transmits the data given from the processing unit 11 to another device and gives the data received from the other device to the processing unit 11.

The display unit 14 is configured by using a liquid crystal display or the like, and displays various images and characters based on the processing of the processing unit 11. In this embodiment, the control device 1 displays the operation status of the robot arm 2 on the display unit 14.

The operation unit 15 receives a user operation and notifies the processing unit 11 of the received operation. For example, the operation unit 15 receives a user's operation with an input device such as a mechanical button or a touch panel provided on the surface of the display unit 14. Further, for example, the operation unit 35 may be an input device such as a mouse and a keyboard, and these input devices may be detachable from the control device 1. In the present embodiment, the user uses the operation unit 15 to set the target value of the liquid amount to be poured from the first container 101 to the second container 102, that is, the target value of the liquid amount to be weighed by the liquid weighing process. be able to.

Further, in the processing unit 11 of the control device 1 according to the present embodiment, the processing unit 11 reads out and executes the control program 12a stored in the storage unit 12, whereby the image acquisition unit 11a, the current angle acquisition unit 11b, and The liquid amount acquisition unit 11c, the next angle acquisition unit 11d, the control processing unit 11e, the learning processing unit 11f, and the like are realized as software-like functional units. It should be noted that these functional units are functional units related to processing of liquid weighing by the robot arm 2, and illustration and description of the other functional units are omitted.

The image acquisition unit 11a performs a process of acquiring an image of the first container 101 captured by the camera 3. In the present embodiment, the camera 3 photographs the entire work performed by the robot arm 2 on the workbench 103. Therefore, the image captured by the camera 3 includes a part or all of the robot arm 2, the first container 101, the second container 102, and the container storage area 104. The image acquisition unit 11a receives the whole image transmitted from the camera 3 at the communication unit 13, and extracts a partial image required for the liquid weighing process from the received whole image. In the present embodiment, the image acquisition unit 11a extracts the region in which the first container 101 is reflected as a partial image from the entire image, and uses the extracted partial image as the image used for the liquid weighing process. In the present embodiment, the camera 3 has a position, a photographing angle, and the like set so that the liquid level of the liquid contained in the first container 101 is reflected while water is being poured into the second container 102. The image acquired by the image acquisition unit 11a includes an image of the liquid level of the liquid contained in the first container 101.

The current angle acquisition unit 11b performs a process of acquiring the current angle of the first container 101 gripped by the robot arm 2. The angle of the first container 101 is an angle of a predetermined axis (for example, a central axis) of the first container 101 with respect to a reference axis (for example, a vertical axis) (see FIG. 8 ). In the present embodiment, the control device 1 appropriately controls the first axis 21 to the sixth axis 26 of the robot arm 2 when moving the first container 101 from the container storage area 104 to a position where water is poured into the second container 102. The first container 101 is conveyed by rotating the first container 101. However, in the present embodiment, the controller 1 fixes the angle without rotating the first shaft 21 to the fifth shaft 25 of the robot arm 2 when the liquid is poured from the first container 101 to the second container 102. Then, only the sixth shaft 26 of the robot arm 2 is rotated to tilt the first container 101. Therefore, the initial angle of the first container 101 in the liquid weighing process is determined by the angles of the first axis 21 to the sixth axis 26 at the end of the carrying process, and the current angle acquisition unit 11b determines the first axis 21 of the robot arm 2. ~ The initial angle of the first container 101 determined by the angle of the sixth axis 26 is acquired. Further, the angle of the first container 101 during the liquid weighing process can be regarded as an increase or decrease of the angle obtained by rotating the sixth axis of the robot arm 2 with respect to the initial angle. The current angle acquisition unit 11b acquires the angle of the sixth axis 26 of the robot arm 2 rotated in the liquid weighing process, and based on the initial angle and the angle of rotation of the sixth axis 26, the current angle of the first container 101 is calculated. By calculating the angle, the current angle of the first container 101 is acquired. The angle of the sixth axis 26 of the robot arm 2 may be calculated, for example, based on the control amount output to the robot arm 2, that is, the rotation amount or the rotation angle when the rotation of the sixth axis 26 is instructed. Alternatively, it may be acquired based on information fed back from the robot arm 2, for example.

The liquid amount acquisition unit 11c performs a process of acquiring the liquid amount of the second container 102 at the present time measured by the measuring device 4. The liquid amount of the liquid measured by the measuring device 4 is a weight or mass expressed in units such as grams (g) or kilograms (kg), or a capacity expressed in units such as liters (L) or cubic meters. Alternatively, the volume or the like may be used. The liquid amount acquisition unit 11c acquires the liquid amount of the second container 102 by receiving the measured liquid amount value transmitted from the measuring device 4 at the communication unit 13.

The image acquisition by the image acquisition unit 11a, the current angle acquisition by the current angle acquisition unit 11b, and the liquid amount acquisition by the liquid amount acquisition unit 11c are repeatedly performed in a predetermined cycle. The period for acquiring these pieces of information can be, for example, several milliseconds to several seconds. The control device 1 may acquire these pieces of information at a cycle determined according to the frame rate of moving image shooting by the camera 3, or may acquire the information at a cycle longer than the cycle determined according to the frame rate. The image acquisition unit 11a, the current angle acquisition unit 11b, and the liquid amount acquisition unit 11c may acquire information at different cycles.

The next angle acquisition unit 11d performs a process of acquiring information (future information) at the next time predicted by the learning model 5 of the storage unit 12 based on information at the current time (latest information). The next angle acquisition unit 11d includes the image of the first container 101 acquired by the image acquisition unit 11a, the current angle of the first container 101 acquired by the current angle acquisition unit 11b, and the current time acquired by the liquid amount acquisition unit 11c. The amount of liquid in the second container 102 and the target amount of liquid to be poured into the second container 102 are input to the learning model 5, and the angle of the first container 101 at the next time output by the learning model 5 is acquired. The next angle acquisition unit 11d repeatedly acquires the next angle at a predetermined cycle.

The control processing unit 11e performs processing for controlling the operation of the robot arm 2. The control processing unit 11e transmits a rotation command specifying the rotation angle or the rotation amount of each of the first axis 21 to the sixth axis 26 of the robot arm 2 to the robot arm 2 so that the robot arm 2 operates. The operation can be controlled. In the present embodiment, the control processing unit 11e fixes the first shaft 21 to the fifth shaft 25 and controls the rotation of the sixth shaft 26 when pouring water from the first container 101 to the second container 102. Water is injected by inclining the first container 101. At this time, the control processing unit 11e controls the rotation of the sixth shaft 26 of the robot arm 2 by using the angle at the next time point acquired by the next angle acquisition unit 11d as the target value of the angle of the first container 101. Note that, for example, when the first container 101 is conveyed from the container storage area 104 to the water pouring position, the control processing unit 11e appropriately rotates the first shaft 21 to the sixth shaft 26 to cause the robot arm 2 to move the first container 101 to the first container 101. It may be transported.

The learning processing unit 11f performs a process of machine learning the learning model 5 used for the liquid weighing process. In the present embodiment, the learning processing unit 11f uses the image of the first container 101, the angle of the first container 101, the target value of the liquid amount and the weighing amount of the second container 102 at a predetermined time point, and the target value at the next time point after the predetermined time point. Machine learning of the learning model 5 is performed using the teacher data in which the image of the one container 101, the angle of the first container 101, the liquid amount of the second container 102, and the target value of the weighing amount are associated with each other. The teacher data may be created in advance by, for example, a designer of the present system, or based on information such as an image and an angle obtained at this time by causing the robot arm 2 to perform a model operation. It may be created by the control device 1.

<Image acquisition process for the first container>
FIG. 4 is a schematic diagram for explaining the image acquisition processing of the first container 101 performed by the control device 1 according to the present embodiment. In the liquid weighing system according to the present embodiment, the control device 1 controls the operation of the robot arm 2 based on the image captured by the camera 3. The camera 3 has its installation position, photographing angle, etc. set so as to photograph the whole work performed by the robot arm 2 on the workbench 103. The entire image 111 obtained by photographing with the camera 3 includes a part of the robot arm 2, the first container 101, the second container 102, the container storage 104, and the like.

In the present embodiment, the control device 1 controls the operation of the robot arm 2 to inject the liquid from the first container 101 to the second container 102, that is, an image required, that is, the learning model 5 for liquid weighing. The image input to is a partial image of the first container 101. In this figure, a partial image showing the first container 101 is shown as a container image 112. The container image 112 shows the liquid level of the liquid contained in the first container 101. Of the entire image 111, a portion other than the container image 112 of the first container 101, for example, a portion such as the second container 102 or the container storage 104 is unnecessary for the liquid weighing process, but a process other than the liquid weighing process, for example, the first container 101. It can be used for carrying processing of one container 101.

Therefore, the control device 1 according to the present embodiment performs image processing of extracting, as the container image 112, the partial image showing the first container 101 from the entire image 111 captured by the camera 3. As a method for the control device 1 to extract the container image 112 from the entire image 111, for example, one of the following two methods can be adopted.

(Container image extraction method 1)
The control device 1 extracts an image of a predetermined area from the entire image 111 taken by the camera 3, and sets the extracted image as the container image 112. The area to be extracted is predetermined by, for example, the designer of the present system, and information such as coordinates that define the area to be extracted is stored in the control device 1. Alternatively, for example, an operator using the present system may be able to set the extraction range of the container image 112 by using the operation unit 15 of the control device 1. This extraction method is suitable when the robot arm 2 tilts the first container 101 to inject water into the second container 102 in a liquid weighing process using the robot arm 2.

(Container image extraction method 2)
The control device 1 performs a process of detecting the first container 101 from the entire image 111 captured by the camera 3. Any method may be adopted for the detection processing of the first container 101 by the control device 1. Since the technology for detecting an object imaged in an image is existing, detailed description thereof will be omitted. The control device 1 uses the learned object detection model that has been machine-learned in advance using the image of the first container 101 to detect the first container 101 in the entire image 111 captured by the camera 3. It can be performed. The control device 1 can extract a rectangular area including the detected first container 101 from the entire image 111 to form a container image 112. This extraction method is suitable when the robot arm 2 may tilt the first container 101 and change the position at which water is poured into the second container 102 in the liquid weighing process using the robot arm 2. Further, the control device 1 may divert the processing result when the object detection is performed in the transportation processing of the first container 101 or the like.

The extraction process of the container image 112 performed by the control device 1 is not limited to the above two methods, and may be performed by any method. The control device 1 extracts the container image 112 showing the first container 101 from the whole image 111 captured by the camera 3, and uses the extracted container image 112 as an input image to the learning model 5. Further, the container image 112 of the first container 101 extracted by the same method is used for the image used as the teacher data in the learning process of the learning model 5. In addition to a camera that captures the entire image, a camera that captures the first container 101 may be provided, and in this case, the control device 1 does not have to perform the process of extracting the container image from the entire image.

<Learning model>
FIG. 5 is a schematic diagram for explaining the configuration of the learning model 5 according to this embodiment. The learning model 5 included in the control device 1 according to the present embodiment includes an image (t) of the first container 101, an angle (t) of the first container 101, and a liquid amount of the second container 102 at a predetermined time (time t). The measured value (t) and the target value of the weighing are accepted as input information, and the image (t+1) of the first container 101, the angle (t+1) of the first container 101, and the second The measured value (t+1) of the liquid amount in the container 102 and the target value for weighing are output. The target value is the same value at time t and time t+1.

The learning model 5 according to the present embodiment includes, for example, an RNN (Recurrent Neural Network) model 51, a CAE (Convolutional Auto Encoder) encoder 52, and a CAE decoder 53. The RNN model 51 is a learning model capable of learning time-series data, and may be, for example, MTRNN (Multi Timescale RNN) or LSTM (Long Short Term Memory). However, the learning model 5 may be configured using a model such as ARIMA (Auto Regressive Integrated Moving Average) or one-dimensional CNN (Convolutional Neural Network) instead of the RNN model 51.

CAE is a learning model that compresses and restores images. However, the learning model 5 may be configured using a model such as an auto encoder (Autoencoder), RBM (Restricted Boltzmann Machine), or principal component analysis (Principal Component Analysis) instead of CAE. The CAE encoder 52 compresses the input image and outputs a feature amount having the feature of the input image. The CAE decoder 53 outputs an image restored from the input feature amount.

The image (t) at the predetermined time (time t) input to the learning model 5 is input to the CAE encoder 52, and the feature amount (t) output at the time t from the encoder 52 is the angle (t) corresponding to the RNN model 51. ), the measured value (t), and the target value. In response to these inputs, the RNN model 51 outputs the feature amount (t+1), the angle (t+1), the measured value (t+1), and the target value at the time point (time t+1) following the predetermined time point. The feature amount (t+1) at the next time output by the RNN model 51 is input to the decoder 53 of the CAE, and the image (t+1) output by the decoder 53 includes the corresponding angle (t+1), the measured value (t+1), and the target value. It becomes the output of the learning model 5.

The control device 1 acquires the angle (t+1) output by the learning model 5 and controls the operation of the robot arm 2. When the controller 1 measures the liquid with the robot arm 2, the image (t+1), the measurement value (t+1), and the target value output by the learning model 5 are unused information. Therefore, the learned learning model 5 stored in the storage unit 12 for the control device 1 to measure the liquid by the robot arm 2 outputs the image (t+1), the measured value (t+1), and the target value. It may not have a function. However, the output function of these pieces of information is necessary when performing machine learning of the learning model 5.

<Learning process>
In the present embodiment, the learning process of the learning model 5 is separately performed by the RNN model 51 and the CAE encoder 52 and decoder 53. FIG. 6 is a schematic diagram for explaining the learning process of the CAE encoder 52 and the decoder 53. In this embodiment, the CAE encoder 52 and the decoder 53 each have a neural network configuration in which a plurality of neurons are connected. The encoder 52 outputs less information from the output layer than the information input to the input layer, and the decoder 53 outputs more information from the output layer than the information input to the input layer. The information input to the input layer of the encoder 52 and the information output from the output layer of the decoder 53 are images, and the amounts of information are the same. The information output from the output layer of the encoder 52 and the information input to the input layer of the decoder 53 are image feature amounts, and the information amounts are equal.

The learning process of the CAE encoder 52 and the decoder 53 is performed by connecting the encoder 52 and the decoder 53 so that the information output from the encoder 52 is input to the decoder 53. The teacher data used in the learning process is the same image as the input image and the output image. The teacher data can be created using the time-series container images 112 captured by the camera 3, but any image may be used. The control device 1 uses such teacher data to perform machine learning on the encoder 52 and the decoder 53 so that the input image and the output image match. As a result, the encoder 52 and the decoder 53 can compress the image into a feature amount and restore the feature amount to the original image. The feature amount output by the encoder 52 has a dimension smaller than that of the input image and represents the feature of the original image without reducing the information amount.

Note that the control device 1 may not perform the learning process of the encoder 52 and the decoder 53, for example, when the existing learned CAE encoder and decoder can be used.

After finishing the learning process of the CAE encoder 52 and the decoder 53, the control device 1 performs the learning process of the learning model 5 including the learned encoder 52 and decoder 53 and the unlearned RNN model 51. At this time, the target of the learning process performed by the control device 1 is the RNN model 51 of the learning model 5, and the encoder 52 and the decoder 53 are outside the target of the learning process. That is, the control device 1 updates the values of the neurons forming the neural network of the RNN model 51 by the learning process, but does not update the values of the neurons forming the neural network of the encoder 52 and the decoder 53.

FIG. 7 is a schematic diagram for explaining the learning process of the learning model 5. In the present embodiment, in order to generate the teacher data used for the learning process of the learning model 5, the control device 1, the robot arm 2, the camera 3, the scale 4, and the like are used, and the robot arm 2 is connected to the first container 101 from the first container 101. The sample operation of pouring the liquid into the second container 102 is performed. In this sample operation, a user (human) operates the robot arm 2 manually. At this time, the control device 1 photographs the sample operation of the robot arm 2 with the camera 3, and at the same time, the angle of the first container 101 (the angle of the sixth axis 26) and the liquid of the second container 102 by the measuring device 4 in the sample operation. The measurement result of the amount is acquired in time series, and the data of the image, the angle, and the liquid amount are stored and stored in the storage unit 12.

It should be noted that any method of manually operating the robot arm 2 by the user may be used. For example, a method using an input device having the same configuration as the robot arm 2 (master-slave method) can be adopted. The user grasps and moves this input device, detects each angle of the input device corresponding to the first axis 21 to the sixth axis 26 of the robot arm 2, and operates the robot arm 2 according to the detected angle. The robot arm 2 can be caused to perform a sample operation of water injection from the first container 101 to the second container 102. Further, for example, a sensor or the like may be attached to the arm of the user, and the sample operation may be performed by reflecting the movement of the user's arm in the operation of the robot arm 2 detected by the sensor.

Further, for example, in the present embodiment, since only the rotation of the sixth shaft 26 of the robot arm 2 is performed to inject water from the first container 101 to the second container 102, a dial switch that operates only the angle of the sixth shaft 26. Alternatively, an input device such as a slide bar may be provided in the operation unit 15 of the control device 1. The user can operate these input devices of the operation unit 15 to perform the sample operation of the robot arm 2. These input devices may be provided in the control device 1 as hardware, for example, and, for example, the control device 1 displays images simulating these input devices on the display unit 34, and accepts a rotation operation or the like by a touch panel or the like. The sample operation of the robot arm 2 may be performed. Further, for example, a device for operating the robot arm 2, that is, a so-called teaching pendant may be used to cause the robot arm 2 to perform a sample operation.

Further, for example, the user does not directly operate the robot arm 2 to perform the sample operation, but the control device 1 causes the robot arm 2 to perform the sample operation according to an arithmetic expression or a table set in advance by the user. May be. The user makes the control device 1 automatically perform the sample operation by creating an arithmetic expression or a table or the like for outputting the angle of the sixth axis 26 in response to the input of time and storing it in the control device 1 in advance. be able to. The arithmetic expression, the table, or the like may be created based on the result of the simulation of the robot arm 2, for example.

By causing the robot arm 2 to perform a series of sample operations of pouring water from the first container 101 to the second container 102, the controller 1 causes the image of the first container 101, the angle of the first container 101, and the second container It is possible to accumulate time-series data with the measured value of the liquid amount 102. Further, the final measured value of the amount of liquid poured into the second container 102 by a series of sample operations can be regarded as the target value of the amount of liquid weighed by this water injection operation.

In FIG. 7, the image (t) of the first container 101 at a predetermined time (time t), the angle (t) of the first container 101, the measured value (t) of the liquid amount of the second container 102, and the liquid The target value of weighing is shown as one data (t). The image (t) of the first container 101 here corresponds to the container image 112. The control device 1 receives the data (t) at the time t, and generates the teacher data in which the data (t+1) at the next time t+1 is associated as the output correct value. The control device 1 sequentially acquires time-series data from the data accumulated for a series of sample operations from the water injection start time point (t=0) to the water injection completion time point (t=Tmax), and the teacher data corresponding to each time point. Can be generated. Note that in the following, the teacher data (t) will be referred to as teacher data in which the data (t) at the time t is input and the data at the time t+1 is used as an output correct value. Teacher data (t) is generated from t=0 to t=Tmax-1.

The control device 1 reads the generated teacher data (t) in time series from t=0 to t=Tmax-1, and applies the read teacher data (t) to the learning model 5 in time series to perform machine learning. By doing so, the parameters of the RNN model 51 of the learning model 5 are repeatedly updated. The learning process of the learning model 5 by the control device 1 is completed by updating the parameters of the RNN model 51 using all the teacher data. In this example, the procedure of generating the teacher data from only one sample operation and performing the learning process is described. However, the control device 1 generates a plurality of sets of teacher data from a plurality of sample operations. Machine learning of the learning model 5 may be performed.

Further, in the present embodiment, the sample operation for generating the teacher data is limited. 8 and 9 are schematic diagrams for explaining the limitation of the sample operation. FIG. 8 illustrates the movement of the first container 101 in the liquid weighing process. The first container 101 is transported from the container storage area 104 by the robot arm 2 to a position (weighing position) for liquid weighing. The state of the first container 101 immediately after being transported to the transport position is shown by the solid line in FIG. 8. This state is the initial state of the first container 101 in liquid weighing, and the angle of the first container 101 in the initial state is Set to 0°. Note that, in FIG. 8, the state in which the first container 101 is upright is the initial state, but this is an example, and the state in which the first container 101 is inclined may be the initial state. However, the initial state of the first container 101 is such that the contained liquid does not leak outside.

The first container 101 gripped by the robot arm 2 is tilted by the rotation of the sixth shaft 26, and water is injected from the first container 101 to the second container 102. FIG. 8 illustrates a state in which the first container 101 is tilted by the angle θ about the sixth axis 26. In the present embodiment, the angle θ of the first container 101 is an angle in which the initial state is 0° and the first container 101 is inclined with respect to the initial state.

FIG. 9 shows a graph of changes in the angle of the first container 101 in the sample operation. The horizontal axis of this graph is the time t, and the vertical axis is the angle θ. In the sample operation, the angle θ of the first container 101 is monotonically increased until the predetermined angle is reached, and then the angle of the first container 101 is monotonically decreased from the predetermined angle to the angle 0°. In the present embodiment, as the teacher data of the learning model 5, data obtained from a sample operation in which the angle of the first container 101 first monotonously increases and then monotonically decreases and then the water injection operation ends is used. By performing the learning process of the learning model using such learning data, the liquid weighing of the robot arm 2 using the learned model 5 learned can be stably performed.

FIG. 10 is a flowchart showing the procedure of the teacher data generation process performed by the control device 1 according to the present embodiment. Although the variable t is used in this flowchart, the variable t can be realized by using a storage area such as a register included in the processing unit 11 of the control device 1, for example. The processing unit 11 of the control device 1 according to the present embodiment causes the robot arm 2 to perform a sample operation of liquid weighing in response to a manual operation by the user (step S1). The processing unit 11 includes an image of the first container 101 (container image 112) extracted from the entire image 111 captured by the camera 3 while the robot arm 2 is performing the sample operation, the angle of the first container 101, and the weighing device 4. The data such as the measured value is acquired and stored in the storage unit 12, and the data is accumulated (step S2). The processing unit 11 determines whether or not the sample operation of the liquid weighing is completed (step S3). When the sample operation is not completed (S3: NO), the processing unit 11 returns the process to step S1 and continues the sample operation and the data accumulation.

When the sample operation is completed (S3: YES), the processing unit 11 initializes the value of the variable t to 0 (step S4). The processing unit 11 acquires the data at the predetermined time t from the accumulated data (step S5). Further, the processing unit 11 acquires the data at the next time t+1 from the accumulated data (step S6). The processing unit 11 receives the data at time t acquired in step S5 as input, and generates teacher data in which the data at time t+1 acquired in step S6 is associated as an output correct value (step S7).

The data at time t acquired in step S5 includes the image (t) of the first container 101 at time t, the angle (t) of the first container at time t, and the second container 102 at time t. The measured value (t) of the liquid amount and the target value of liquid weighing are included. The data at time t+1 acquired in step S6 includes the image (t+1) of the first container 101 at time t+1, the angle (t+1) of the first container at time t+1, and the liquid in the second container 102 at time t+1. The measured value of volume (t+1) and the target value of liquid weighing are included. The target value of both data is set to the measured value (Tmax) of the liquid amount in the second container 102 at the end of the sample operation.

After that, the processing unit 11 adds 1 to the value of the variable t (step S8). The processing unit 11 determines whether or not the value of the variable t is the maximum value Tmax (step S9). When the value of the variable t is not Tmax (S9: NO), the processing unit 11 returns the process to step S5 and continues the generation of the teacher data. When the value of the variable t is Tmax (S9: YES), the processing unit 11 ends the teacher data generation process.

FIG. 11 is a flowchart showing the procedure of the learning process of the learning model 5 performed by the control device 1 according to the present embodiment. Although the variable t is used in this flowchart, the variable t can be realized by using a storage area such as a register included in the processing unit 11 of the control device 1, for example. The learning processing unit 11f of the processing unit 11 of the control device 1 according to the present embodiment first initializes the value of the variable t to 0 (step S21). The learning processing unit 11f acquires the teacher data corresponding to the predetermined time t from the storage unit 12 (step S22). The learning processing unit 11f uses the image at time t, the angle, the measured value, and the target value included in the acquired teacher data as the input of the learning model 5, and sets the image at the time t+1, the angle, the measured value, and the target value of the learning model 5. As the correct value of the output, the learning model 5 updates the parameters of the RNN model 51 so that the input/output relationship is satisfied (step S23). The parameters of the RNN model 51 may be updated collectively for a plurality of teacher data.

After that, the learning processing unit 11f adds 1 to the value of the variable t (step S24). The learning processing unit 11f determines whether the value of the variable t is the maximum value Tmax (step S25). When the value of the variable t is not Tmax (S25: NO), the learning processing unit 11f returns the process to step S22 and continues the learning of the learning model 5. When the value of the variable t is Tmax (S25: YES), the processing unit 11 ends the learning process.

<Liquid weighing processing>
After the learning process of the learning model 5 is completed, the control device 1 can use the learned learning model 5 to perform the liquid weighing process by the robot arm 2. In the present embodiment, the robot arm 2 grips the first container 101 from the state where the first container 101 is placed in the container storage area 104 of the work table 103 and the second container 102 is placed on the weighing device 4. Then, the first container 101 is tilted to inject water into the second container 102 until the target value of water is completed and the robot arm 2 places the first container 101 in the container storage area 104. The operation will be described. Note that the target value of the weighing is assumed to be preset by the user using the operation unit 15 of the control device 1, for example.

FIG. 12 is a flowchart showing a schematic procedure of the liquid weighing process performed by the control device 1 according to the present embodiment. The processing unit 11 of the control device 1 according to the present embodiment first controls the operation of the robot arm 2 so that the robot arm 2 holds the first container 101 placed in the container storage area 104 of the workbench 103 ( Step S31). The operation of the robot arm 2 for gripping the first container 101 may be controlled by any method.

Next, the processing unit 11 conveys the first container 101 held in step S31 to the weighing position where the liquid is weighed (step S32). The transfer of the first container 101 by the robot arm 2 may be controlled by any method. The weighing position, which is the end point of the transport operation, may be, for example, a preset fixed position, or may be a position that varies depending on, for example, the position of the second container 102.

Next, the processing unit 11 performs an operation of injecting the liquid contained in the first container 101 into the second container 102, and performs a weighing process for weighing a predetermined amount of liquid (step S33). In the weighing process, the processing unit 11 rotates only the sixth shaft 26 of the robot arm 2 to tilt the first container 101 and injects water from the first container 101 into the second container 102. Details of the weighing process will be described later.

After pouring a predetermined amount of liquid from the first container 101 into the second container 102 and completing the weighing process, the processing unit 11 controls the robot arm 2 to transfer the first container 101 from the weighing position to the container storage area 104. Then (step S34), the process ends. The transfer of the first container 101 by the robot arm 2 may be controlled by any method. The destination of the first container 101 after the weighing process may be a different place from the container storage area 104 where the first container 101 was initially placed.

FIG. 13 is a flowchart showing a detailed procedure of the liquid weighing process performed by the control device 1 according to the present embodiment, which is a detailed procedure of the weighing process performed in step S33 of FIG. The image acquisition unit 11a of the processing unit 11 of the control device 1 according to the present embodiment acquires the image of the first container 101 captured by the camera 3 (step S41). The image acquired in step S41 is a container image obtained by extracting the portion of the first container 101 from the entire image. Further, the current angle acquisition unit 11b of the processing unit 11 acquires the angle of the first container 101 based on the angles from the first axis 21 to the sixth axis 26 of the robot arm 2 (step S42). Further, the liquid amount acquisition unit 11c of the processing unit 11 acquires the measured value of the liquid amount of the second container 102 measured by the measuring device 4 (step S43).

The next angle acquisition unit 11d of the processing unit 11 includes the image acquired in step S41, the angle acquired in step S42, the measured value of the liquid amount acquired in step S43, and the weighing target preset by the user. The current data including the value is input to the learned learning model 5 stored in the storage unit 12 (step S44). The next angle acquisition unit 11d acquires the angle of the next time point of the first container 101 from the data output by the learning model 5 in response to the data input in step S44 (step S45).

The control processing unit 11e of the processing unit 11 determines whether or not the angle at the next time point acquired in step S45 is less than or equal to a preset upper limit angle (step S46). When the angle at the next time is less than or equal to the upper limit angle (S47: YES), the control processing unit 11e changes the angle of the first container 101 to the angle at the next time acquired in step S45. The rotation of the sixth shaft 26 is controlled (step S47), and the process proceeds to step S49. When the angle at the next time point exceeds the upper limit angle (S47: NO), the control processing unit 11e maintains the angle of the sixth axis 26 of the robot arm 2 in order to maintain the current angle of the first container 101 (( The process proceeds to step S48) and step S49.

Next, the processing unit 11 determines whether or not the liquid weighing operation is completed (step S49). Any method may be used to determine whether or not the liquid weighing process is completed. For example, the processing unit 11 can determine that the liquid weighing is completed when the angle of the first container 101 returns to the initial angle (that is, the angle of 0°). Further, for example, the processing unit 11 may determine whether or not the discharge of the liquid from the first container 101 has ended, based on the image captured by the camera 3, to determine the end of liquid weighing. When the liquid weighing process is not completed (S49: NO), the processing unit 11 returns the process to step S41. When the liquid weighing process is completed (S49: YES), the processing unit 11 returns the process to the flowchart shown in FIG.

<Summary>
The liquid weighing system according to the present embodiment having the above configuration measures the liquid by injecting a predetermined amount of the liquid from the first container 101 held by the robot arm 2 into the second container 102. The control device 1 that controls the operation of the robot arm 2 receives the image of the first container 101 at the first time point (time t), the angle of the first container 101, and the target values of the liquid amount and the weighing amount of the second container 102 as inputs. The learning model 5, which has been machine-accepted and output to the angle of the first container 101 at the second time point (time t+1), is stored in the storage unit 12. The control device 1 acquires an image of the first container 101 captured by the camera 3 at the current time (first time), acquires the angle of the first container 101 at the current time, and determines the liquid amount of the second container 102 at the current time. get. The control device 1 inputs the acquired image, angle, and liquid amount into the learning model 5, and acquires the angle of the first container 101 at the next time point (second time point) output by the learning model 5. The control device 1 controls the angle of the first container 101 held by the robot arm 2 according to the acquired angle at the next time. As a result, the control device 1 can control the robot arm 2 to inject a predetermined amount of liquid from the first container 101 into the second container 102, and accurately measure the liquid using the robot arm 2. Can be expected.

Further, the control device 1 according to the present embodiment fixes the first shaft 21 to the fifth shaft 25 of the robot arm 2 when the first container 101 is tilted and water is poured into the second container 102, and the first container 101 is fixed. The rotation of the sixth shaft 26 closest to the holding position of is controlled to control the angle of the first container 101. The control device 1 changes only the angle of the first container 101 to the second container 102 without changing the parameters other than the angle (for example, the position in the vertical direction, the horizontal direction, the front-back direction, etc.) of the first container 101. Water injection can be performed. This can be expected to facilitate the control of the robot arm 2 by the control device 1 and the learning model 5.

Further, the control device 1 according to the present embodiment uses the first container based on the entire image taken by the camera 3 so as to include the robot arm 2, the first container 101, the second container 102, the work table 103, the container storage space 104, and the like. The image 101 is acquired and used as input data to the learning model 5. The control device 1 may extract a predetermined region from the entire image as a container image, or may detect the first container 101 shown in the entire image and extract the container image. As a result, noise (images other than the first container 101) included in the learned image is reduced, and the learning accuracy of the learning model 5 is improved. As a result, it can be expected that the angle of the first container 101 using the learning model 5 is accurately controlled.

Further, in the present embodiment, when the control device 1 controls the robot arm 2 and tilts the first container 101, the upper limit angle is set for the angle of the first container 101, and the upper limit angle is not exceeded. The controller 1 controls the angle of the first container 101. This can prevent the angle of the first container 101 from becoming too large, and can be expected to perform the weighing operation stably.

Further, in the present embodiment, the position, the shooting angle, etc. of the camera 3 are set so that the image captured by the camera 3 includes the image of the liquid surface of the liquid contained in the first container 101. As a result, the learning process of the learning model 5 including the information such as the viscosity of the liquid contained in the first container 101 becomes possible, and the accuracy of liquid weighing can be expected to improve.

Further, in the present embodiment, the teacher data used for the learning process of the learning model 5 is generated based on the data obtained by the sample operation of the robot arm 2. In the sample operation, the robot arm 2 monotonically increases the angle of the first container 101 from the reference angle to the predetermined angle, and then monotonically decreases from the predetermined angle to the reference angle to change the first container 101 to the second container 102. Complete water injection. The control device 1 generates teacher data based on the time-series image, the angle, and the liquid amount in this sample operation. By using the learning model 5 that is machine-learned using this teacher data, the control device 1 monotonically increases the angle of the first container 101 from the reference angle to the predetermined angle, and then monotonically increases from the predetermined angle to the reference angle. The operation of the robot arm 2 can be controlled so that the water injection is completed by decreasing the amount.

The embodiments disclosed this time are to be considered as illustrative in all points and not restrictive. The scope of the present invention is shown not by the above meaning but by the scope of the claims, and is intended to include meanings equivalent to the scope of the claims and all modifications within the scope.

DESCRIPTION OF SYMBOLS 1 Control device 2 Robot arm 3 Camera 4 Measuring instrument 5 Learning model 11 Processing part 11a Image acquisition part 11b Current angle acquisition part 11c Liquid amount acquisition part 11d Next angle acquisition part 11e Control processing part 11f Learning processing part 12 Storage part 13 Communication part 14 display part 15 operation part 20a 1st base part 20b 2nd base part 20c 1st arm part 20d 2nd arm part 20e 3rd arm part 21 1st axis 22 2nd axis 23 3rd axis 24 4th axis 25 5th Axis 26 Sixth Axis 51 RNN Model 52 Encoder 53 Decoder 99 Recording Medium 101 First Container 102 Second Container 103 Workbench 104 Container Storage Area 111 Overall Image 112 Container Image

Claims (14)

A liquid weighing method for pouring a predetermined amount of liquid from a first container held by a robot arm into a second container,
A learning model machine-learned to accept the image of the first container at the first time point, the angle of the first container, and the liquid amount of the second container as inputs, and output the angle of the first container at the second time point. Using
A control device for controlling the operation of the robot arm,
Acquiring an image including the first container taken at a first time point,
Obtaining the angle of the first container at the first time point,
Obtaining the liquid amount of the second container at the first time point,
The acquired image, angle, and liquid amount are input to the learning model to acquire the angle at the second time point output by the learning model,
Controlling the angle of the first container held by the robot arm according to the acquired angle at the second time point;
Liquid weighing method. The control device controls the operation of the robot arm such that parameters other than the angle of the first container are not changed.
The liquid weighing method according to claim 1. The robot arm has a plurality of rotation axes that can be controlled by the control device,
The control device controls the rotation of one of the plurality of rotation shafts, fixes the rotation shafts other than the rotation shaft, and controls the operation of the robot arm.
The liquid weighing method according to claim 2. The control device controls rotation of one rotation shaft closest to a holding position of the first container to control an angle of the first container held by the robot arm,
The liquid weighing method according to claim 3. The control device acquires an image of the first container from an entire image captured including the robot arm, the first container, and the second container, and inputs the image to the learning model.
The liquid weighing method according to any one of claims 1 to 4. The control device is
Detecting the first container from the overall image,
Acquiring an image of the detected first container from the overall image,
The liquid weighing method according to claim 5. An upper limit is set for the angle of the first container,
The liquid weighing method according to any one of claims 1 to 6. The image including the first container includes an image of the liquid surface of the liquid contained in the first container,
The liquid weighing method according to any one of claims 1 to 7. The control device monotonically increases the angle of the first container from a reference angle to a predetermined angle, and then monotonically decreases from the predetermined angle to the reference angle, thereby injecting water from the first container to the second container. To control the angle of the first container held by the robot arm,
The liquid weighing method according to any one of claims 1 to 8. A control device comprising a control unit for controlling operation of a robot arm, the control device performing control for pouring a predetermined amount of liquid from a first container held by the robot arm into a second container,
A learning model machine-learned to accept the image of the first container at the first time point, the angle of the first container, and the liquid amount of the second container as inputs, and output the angle of the first container at the second time point. A storage unit for storing
An image acquisition unit that acquires an image including the first container captured at a first time point;
A current angle acquisition unit that acquires the angle of the first container at the first time point,
A liquid amount acquisition unit that acquires the liquid amount of the second container at the first time point,
The image acquired by the image acquisition unit, the current angle acquired by the current angle acquisition unit, and the liquid amount acquired by the liquid amount acquisition unit are input to the learning model stored in the storage unit, and the learning model is acquired. And a next angle acquisition unit that acquires the angle at the second time point output by
The control unit controls the angle of the first container held by the robot arm according to the angle at the second time point acquired by the next angle acquisition unit,
Control device. A computer that controls the operation of the robot arm that injects a predetermined amount of liquid from the first container to the second container,
Acquiring an image including the first container taken at a first time point,
Obtaining the angle of the first container at the first time point,
Obtaining the liquid amount of the second container at the first time point,
A learning model machine-learned to accept the image of the first container at the first time point, the angle of the first container, and the liquid amount of the second container as inputs, and output the angle of the first container at the second time point. In contrast, by inputting the acquired image, angle and liquid amount at the first time point, the angle at the second time point output by the learning model is acquired,
Controlling the angle of the first container held by the robot arm according to the acquired angle at the second time point;
Computer program. Acquiring a time-series image including the first container, which is photographed when pouring a predetermined amount of liquid from the first container held by the robot arm into the second container,
The robot arm acquires a time series angle of the first container when the predetermined amount of liquid is poured from the first container to the second container,
The robot arm acquires a time-series liquid amount of the second container when the predetermined amount of liquid is poured from the first container to the second container,
Teacher data in which the acquired image, angle, and liquid amount at the first time point are associated with the image, angle, and liquid amount at the second time point,
Machine learning is performed using the generated teacher data for a learning model that receives the image, angle, and liquid amount at the first time point as input, and outputs the image, angle, and liquid amount at the second time point,
The image of the first container at the first time point, the angle of the first container, and the liquid amount of the second container are accepted as inputs, and the image of the first container at the second time point, the angle of the first container, and the Generate a learning model that outputs the amount of liquid in two containers,
Learning method. The image of the teacher data includes an image of the liquid level of the liquid contained in the first container,
The learning method according to claim 12. The robot arm monotonically increases the angle of the first container from a reference angle to a predetermined angle, and then monotonically decreases from the predetermined angle to the reference angle, thereby injecting water from the first container to the second container. Generating the teacher data based on the time-series image, the angle, and the liquid amount when completed.
The learning method according to claim 12 or 13.

Images