JP2021121306A - Automatic dish washing system, automatic dish washing method, automatic dish washing program, and storage medium - Google Patents

Abstract

To provide an automatic dish washing system capable of picking up dishes by holding various parts according to types of dishes with a robot arm, and storing the dishes in a washing rack, an automatic dish washing method, an automatic dish washing program, and a storage medium.SOLUTION: An automatic dish washing system 100 includes: a robot arm 110 having a hand 112 at the tip of an arm 111; a tray arranging unit 120 where trays are arranged; a rack arranging unit 140 where a washing rack 142 for storing dishes is arranged; a dish information acquisition unit 160 for acquiring image information on the dishes on the tray arranged in the tray arranging unit 120; and a dish conveyance control unit 171 for controlling the robot arm 110 so as to calculate a hand holding part of the dish that the hand 112 holds by sandwiching or adsorbing, pick up the dish from the tray arranging unit 120 while holding the hand holding part, and store the dish in the washing rack.SELECTED DRAWING: Figure 2

Description

The present invention relates to an automatic dishwashing system for automatically washing dishes and the like, an automatic dishwashing method, an automatic dishwashing program, and a storage medium.

In recent years, due to the shortage of employees in restaurants, there has been a demand for automating the processes of operations and services in restaurants. There is also a demand for automation of dishwashing, and techniques such as Patent Document 1 have been proposed in the past.

The dishwashing system described in Patent Document 1 includes a dishwasher for washing dishes stored in a washing rack in a washing room, a loading stand for mounting a washing rack carried into the washing room, and a loading table. It is provided with a carrying-out table for mounting a washing rack carried out from the washing room and a robot device having a robot arm mechanism. The carry-in table and the dishwasher are arranged within the movable range of the robot arm mechanism. The robot arm mechanism transfers the washing rack from the loading platform to the washing room and operates the dishwasher.

WO 2018/034252

In the dishwashing system described in Patent Document 1, it is difficult to hold and pick up a portion other than the bottom surface of the tableware by a robot arm.
Further, in the dishwashing system described in Patent Document 1, it is not described that the used dishes are automatically stored in the washing rack, and it cannot be said that the dishes are automated.

An object of the present invention is an automatic tableware washing system capable of picking up dishes by holding a part other than the bottom surface of the dishes by a robot arm and automatically storing the dishes picked up by the robot arm in a washing rack. , An automatic dishwashing method, an automatic dishwashing program, and a storage medium.

The above object of the present invention can be achieved by the following configuration. That is, the automatic dishwashing system according to the first aspect of the present invention is
A robot arm with a hand at the tip of the arm,
The tray placement section where the tray on which the tableware is placed is placed,
A rack arrangement part in which a washing rack for accommodating tableware picked up by the robot arm is arranged, and
A tableware information acquisition unit that acquires image information of tableware on the tray arranged in the tray arrangement unit, and
Based on the image information of the tableware acquired by the tableware information acquisition unit, the hand holding portion of the tableware held by the hand is recognized, the hand holding portion is held, and the tableware is picked up from the tray arrangement unit and picked up. A tableware transport control unit that controls the robot arm so that the tableware is housed in the washing rack.
It is characterized by having.

According to the present invention, the tableware can be picked up by holding a part other than the bottom surface of the tableware by the robot arm, and the tableware picked up by the robot arm can be automatically stored in the washing rack. , Tableware automatic cleaning methods, tableware automatic cleaning programs, and storage media can be provided.

It is a perspective view of the tableware automatic washing system which concerns on Embodiment 1 of this invention. It is a block diagram which shows schematicly about the whole structure including the part about the control and drive of the tableware automatic washing system 100. It is a top view which shows the periphery of the hand of a robot arm. It is the top view which shows the periphery of the hand of the robot arm in the state which fixed the stereo camera to the hand. It is a figure for demonstrating the hand holding part of the tableware which the hand holds and holds for picking up by a robot arm. It is a figure which showed the output result of image recognition by AI. It is a figure for demonstrating the process of calculating the storage position of the tableware in the washing rack from the image information of the tableware picked up by the robot arm and the image information of the washing rack. FIG. 5 is a block diagram schematically showing an overall configuration including a part related to control and driving of an automatic tableware washing system according to a second embodiment of the present invention.

Hereinafter, the tableware automatic washing system, the tableware automatic washing system, the tableware automatic washing method, the tableware automatic washing program, and the storage medium according to the embodiment of the present invention will be described with reference to the drawings. However, the embodiments shown below exemplify an automatic dishwashing system for embodying the technical idea of the present invention, and do not specify the present invention as these, but are included in the scope of claims. It is equally applicable to those of the embodiment of.

[Embodiment 1]
The automatic dishwashing system according to the first embodiment of the present invention will be described with reference to FIGS. 1 to 6.

FIG. 1 is a perspective view of the tableware automatic washing system 100 according to the first embodiment of the present invention. FIG. 2 is a block diagram schematically showing an overall configuration including a part related to control and driving of the automatic dishwashing system 100. FIG. 3 is a perspective view showing the periphery of the hand 112 of the robot arm 110. FIG. 4 is a diagram for explaining a hand holding portion 180a of the tableware 180 held by the hand 112 for being picked up by the robot arm 110. FIG. 5 is a diagram showing the output result of image recognition by AI. FIG. 6 is a diagram for explaining a process of calculating the storage position of the tableware 180 in the washing rack 142 from the information of the tableware 180 picked up by the robot arm 110 and the information of the empty space of the washing rack 142. ..

The tableware automatic washing system 100 of the present embodiment includes a robot arm 110 having a hand 112 at the tip of the arm 111, a tray arranging unit 120 on which a tray 182 on which used tableware 180 is placed is placed, and a robot arm 110. A rack in which a pre-washing unit 130 for pre-washing the tableware 180 picked up from the tray 182 arranged in the tray arranging unit 120 and a washing rack 142 for accommodating the tableware 180 picked up by the robot arm 110 for washing are arranged. The arrangement unit 140, the tableware cleaning unit 150 for cleaning the tableware 180 housed in the cleaning rack 142, and the tableware information acquisition unit 160 for acquiring image information of the tableware 180 on the tray 182 arranged in the tray arrangement unit 120. , A control device 170 that controls the operation of the entire automatic dishwashing system 100.
The tableware 180 includes, for example, Western tableware, Japanese tableware, shape, etc., for example, plates, balls, bowls, bowls, bowls, etc. for serving food, cups, cups, glasses, tea bowls, etc. , Sake bowls, chopsticks, forks, knives, spoons, madler, rangers, etc. that handle food, pots, frying pans, cutting boards, kitchen utensils, knives, balls, cups, containers, frying, spatula, ladle Etc., Includes all tools and utensils used for meals.

As the robot arm 110, for example, a 6-axis articulated robot arm is used, and the hand 112 is provided at the tip of the arm 111. As long as the robot arm 110 is an articulated type, a robot arm 110 other than the 6-axis robot arm 110 may be used.
The robot arm 110 has a robot control unit 110a that controls the operation of the robot arm 110, and the robot control unit 110a is connected to the control device 170.

The hand 112 has a pair of holding pieces 113, and a part of the tableware 180 is held by the pair of holding pieces 113.
In the pair of holding pieces 113, the tip of one holding piece 113 is bent outward, and the tip of the other holding piece 113 is bent inward. The pair of holding pieces 113 have a flat shape from a bent portion in which the holding surface 113a, which is a surface on the side of holding the tableware 180, is bent in an R shape toward the tip.
The hand 112 has a water-resistant and oil-resistant silicon material or the like attached to the inside of the tip portion to stably hold the tableware 180.
In this hand 112, when the pair of holding pieces 113 are brought close to each other, the holding surfaces 113a on the holding side of the tip of each other face each other in parallel, and the opening of the tableware 180 at the tips of the pair of holding pieces 113. The periphery of the edge portion can be sandwiched between the planar sandwiching surfaces 113a facing each other in parallel.

Further, when the hand 112 is oriented vertically downward, the tips of the pair of holding pieces 113 are in a lateral direction, so that the water jetted from the rotating nozzle of the dishwashing unit 150 is discharged. It is possible to facilitate the arrangement of the tableware 180 in a posture in which the tableware 180 is easily hit, that is, in a posture in which the edge surface of the open end is tilted downward or in a posture in which the tableware 180 is turned down.

In order to grip chopsticks and cutlery with the hand 112, for example, a bifurcated hand portion or an air gripper may be used to grip the chopsticks and cutlery. Further, it is also possible to prepare a plurality of types of hands 112 and use the hands properly according to the object to be gripped.

The tray arranging unit 120 includes a receiving tray arranging unit 121 on which a plurality of trays 182 on which used tableware 180 is placed are received, and one tray 182 from the receiving tray arranging unit 121 is provided with the tableware 180 by the robot arm 110. It has a tableware picking tray arranging portion 122, which is transferred and arranged at a position to be picked up.

The receiving tray arranging unit 121 has a tray mounting table 121a capable of arranging a plurality of trays 182 in a row and arranging the trays 182 in a row and moving the trays forward in the arrangement direction of the trays 182.
The tray mounting table 121a is inclined downward toward the front side in the arrangement direction of the tray 182, and the tray 182 is automatically moved forward by using a spring mechanism (not shown). There is.

A guide member 123 against which the tray 182 at the foremost position is abutted is arranged at the front end of the tray mounting table 121a. The guide member 123 is provided so as to be slidably movable between the tray mounting table 121a and the tableware picking tray arranging portion 122 so that the guide member 123 can be pulled into the tableware picking tray arranging portion 122 by the robot arm 110. There is.
One tray 182 at the foremost position, which is abutted against the guide member 123, is pushed by the guide member 123 which is slid and moved by the robot arm 110, and is moved together with the guide member 123, whereby the tableware picking tray arranging portion 122 It is designed to be placed in.

The tableware picking tray arranging unit 122 has a tray mounting table 122a that is driven up and down by an elevating mechanism (not shown) driven by an elevating drive unit 122b connected to the control device 170. A plurality of the tableware picking tray arranging portions 122 can be stacked and placed on the tray placing table 122a.
The tableware pick-up tray arranging unit 122 is newly moved to the uppermost position when all the used tableware 180 is moved from the tray 182 transferred from the receiving tray arranging unit 121 to the washing rack 142 by the robot arm 110. The tray mounting table 122a is lowered so that one tray 182 can be received.
In such a tableware pick-up tray arranging unit 122, a plurality of trays 182 received by the receiving tray arranging unit 121 can be stacked and arranged so that they can be collectively collected as used trays 182. It has become.

The pre-washing unit 130 has a sink 130a, and moves the tableware 180 picked up by the robot arm 110 by the robot arm 110 while pressing it against a brush (not shown) arranged in water stored in the sink 130a. This is the part where the tableware 180 is pre-washed.

The rack arranging unit 140 stocks a plurality of cleaning racks 142 in the vertical direction so that the tableware 180 can be stored in the cleaning rack 142 at the uppermost position by the robot arm 110, and the storage of the tableware 180 is completed. When the washing rack 142 in the uppermost position is moved to the dishwashing unit 150, the washing rack 142 arranged below can be raised to the uppermost position.
The rack arranging unit 140 has an elevating mechanism 141 for moving a plurality of stocked cleaning racks 142 up and down by an elevating drive unit 141a connected to the control device 170, and cleaning stocked downward by the elevating mechanism 141. The rack 142 is sequentially supplied to the uppermost position.
The washing rack 142 is moved from the rack arrangement unit 140 to the dishwashing unit 150 by the rack delivery mechanism 143. The rack delivery mechanism 143 is driven by a drive unit 143a connected to the control device 170.

The tableware washing unit 150 removes dirt adhering to the tableware 180 housed in the washing rack 142 by using, for example, water jetted from a rotating nozzle and an automatically injected detergent, and then dries the tableware 180. Is what you do.
The dishwashing unit 150 has a dishwashing control unit 150a that controls a series of operations related to washing the tableware 180 from washing to drying, and a control device 170 is connected to the dishwashing control unit 150a.
When the dishwashing unit 150 completes the washing of the tableware 180, the washing rack 142 is sent out from the dishwashing unit 150, and for example, each tableware 180 is taken out from the washing rack 142 by a robot arm (not shown) and washed. It is stored as tableware 180 in a storage container (not shown).

The tableware information acquisition unit 160 is realized by, for example, a stereo camera 160 capable of acquiring two-dimensional image information and / or three-dimensional image information of the tableware 180 by a three-dimensional point cloud.
In the present embodiment, an example in which a color camera is used as the stereo camera will be described, but the present invention is not limited to the color camera, and a monochrome camera may be used.
In this embodiment, the stereo camera 160 is fixed to the hand 112. The stereo camera 160 can acquire image information by directing the optical axis of the stereo camera 160 in an arbitrary direction by moving the robot arm 110. That is, by moving the robot arm, it is possible to shoot from various angles, and the shooting position and angle can be accurately obtained from the robot side.
Therefore, in the stereo camera 160, the hand 112 is directed to the tray 182 arranged in the tray arrangement portion 120 or the washing rack 142 arranged in the rack arrangement portion 140, so that the tableware 180 and the rack on the tray 182 are directed. Acquires two-dimensional information and / or three-dimensional image information of the accommodating portion of the cleaning rack 142 arranged in the arrangement portion 140.

In the present embodiment, the use of the stereo camera 160 as the tableware information acquisition unit 160 is illustrated, but the tableware information acquisition unit 160 is not limited to the stereo camera, and other various types of cameras and image acquisition means. including. Further, in the present embodiment, an example in which one camera is provided will be described, but the present application is not limited to this, and a plurality of cameras may be used. For example, a mobile camera provided on the hand and another fixed camera can be used. Using multiple cameras makes it easier to recognize multiple stacked dishes.

In addition to the stereo camera method described above, as a method of obtaining a three-dimensional point cloud of the tableware 180, 1) a pattern such as a laser beam or an LED is irradiated to photograph how the pattern is reflected, and the three-dimensional point cloud is obtained. Method of taking (Structured Light method), 2) Method of irradiating light such as laser light or LED and measuring the time until it is reflected and returned (Time of Flight method), and 3) Focusing of the image A method of searching for a position and knowing depth information (focusing method) or the like can also be used.

The control device 170 can be realized by, for example, a personal computer, a tablet terminal, or the like. The control device 170 is connected to each part of the automatic tableware washing system 100 as described above, and after the tray 182 on which the used tableware 180 is placed is received by the receiving tray arrangement part 121, it is washed by the dishwashing part 150. It controls a series of dishwashing processes by the automatic dishwashing system 100 until it is completed.
The control device 170 has a tableware transport control unit 171. The tableware transport control unit 171 is installed as software, for example, in a storage unit as a storage medium (not shown) of the control device 170.

The tableware transport control unit 171 is arranged in the tableware pick-up tray arrangement unit 122 based on the two-dimensional image information of the tableware 180 acquired by the stereo camera 160 and / or the three-dimensional image information of the tableware 180 by the three-dimensional point group. The hand holding portion 180a of the tableware 180 held by the hand 112 to be picked up by the robot arm 110 in order to pick up the tableware 180 on the tray 182 is calculated, and the hand holding portion 180a is held and the tableware 180 is held from the tray arrangement portion 120. The robot arm 110 is controlled so as to accommodate the picked up tableware 180 in the washing rack 142.
The two-dimensional image information is information including the arrangement, shape, size, and color of the tableware 180. Further, the three-dimensional image information is information including the shape of the tableware, the height of the edge, and the color.

The tableware transport control unit 171 recognizes the arrangement, shape, size, color, and height of the edge of the tableware 180 from the two-dimensional image information or the three-dimensional image information acquired by the stereo camera 160, and sandwiches the tableware 180. And calculate the hand holding part to be held.
As shown in FIG. 4, the hand holding portion 180a is, for example, an area around the open end of the tableware 180.

Since the tableware transport control unit 171 is learned from the image information of the general tableware 180, it is possible to recognize the tableware 180 having various shapes, sizes, colors, and edge heights that are not model data. ..
That is, the tableware transport control unit 171 uses machine learning to arrange, shape, and size the tableware 180 even when the two-dimensional image information and / or the three-dimensional image information of the tableware 180 is not registered (modelless). The color and the height of the edge can be estimated, and by calculating the hand holding portion based on this estimation, it is possible to pick up the tableware 180 using the robot arm 110.

Since the posture of the robot arm 110 is limited, it may not be possible to move the hand 112 at the tip of the arm in the shortest distance. The correction amount of the trajectory of the arm tip of the robot arm 110 and which tableware 180 on the tray 182 should be selected as the target so that the next tableware 180 can be smoothly picked up following the operation of picking up the current tableware 180. It is also possible to learn to optimize. For example, by using the learned AI, the robot arm 110 can quickly pick up the tableware 180 with a smooth movement like a skilled person from the initial introduction of the automatic tableware washing system 100. For example, it is possible to put a correction amount in the trajectory of the robot arm 110 when it is picked up and optimize the correction amount by reinforcement learning in the direction in which the picking up is successful, or supervised learning by DNN (deep neural network). It is effective and contributes to speeding up the robot arm 110.

There are innumerable postures of the robot arm 110 for picking up the tableware 180 by the robot arm 110, but if the posture is not properly selected, the robot arm 110 will stop because it is too close to the movable limit. Therefore, using the operability, which is an index of the mobility of the robot arm 110, and the proximity to the movable limit, the tableware 180 is selected from among several posture candidates in which the robot arm 110 is easy to move. take up.

Further, the tableware transport control unit 171 calculates an empty space of the cleaning rack 142 based on the image information in the accommodating unit of the cleaning rack 142 acquired by the stereo camera 160, and a robot for accommodating the tableware in the cleaning rack 142. The storage position of the tableware 180 in the washing rack 142 is calculated from the information of the tableware 180 picked up by the arm 110 and the information of the empty space of the washing rack 142, and the robot arm 110 sets the calculated storage position of the tableware 180 in the washing rack 142. Accommodates tableware 180.
More specifically, the tableware transport control unit 171 recognizes the empty space seen from the upper part of the washing rack 142 by, for example, a square pattern P, and the empty space of one or more patterns P serving as the empty space of the washing rack 142. A pattern P in which the tableware 180 picked up by the robot arm 110 is accommodated is selected from the inside, and the robot arm 110 is controlled so as to arrange the tableware 180 at the position of the selected pattern P.

The tableware transport control unit 171 recognizes the shape of the tableware 180 from the two-dimensional image information and / or the three-dimensional image information of the tableware 180 currently taken up by the robot arm 110, and the shape of the tableware 180 and the pattern of one or more empty spaces. The degree of coincidence with P is evaluated, and the robot arm 110 is controlled so that the tableware 180 is arranged at the position of the pattern P having a high degree of coincidence.

The entire inside of the washing rack 142 is image-recognized, the tableware already stored is image-recognized, and the empty space is also grasped. In order to rationally pack the tableware in this empty space, not only the type of tableware currently held by the robot arm, but also the next tableware, the next tableware, etc. Use the information.
Also, when packing various dishes in the washing rack 142, "Don't stack the bowl immediately after the glass cup" (so that the glass cup does not break when hitting the bowl), or "If possible, the same type of dish should be used. There is a rule such as "If you put them together for each, the volumetric efficiency is good."
By using the subsequent tableware information, the rules for packing the dishes in the washing rack 142, and the like, the method of packing the dishes in the washing rack 142 can be further optimized by training the AI by machine learning.

Regarding the arrangement for packing dishes in the washing rack 142, AI has the following specifications.
(1) By recognizing the tableware object in the washing rack 142, the empty space area including the determination of the position and size in the washing rack 142 is grasped.
(2) Attitude control at the time of placement is performed based on the size, height, and shape information of the tableware being held.
(3) The tableware place rule follows the rule base because there are restrictions on the order of tableware on the pick side.

Next, a series of operations of the automatic dishwashing system 100 in the dishwashing process will be described.
First, the robot arm 110 pulls one tray 182 on which the used tableware 180 is placed from the receiving tray arranging unit 121 into the tableware picking tray arranging unit 122.
When the robot arm 110 pulls the guide member 123 toward the tableware picking tray arranging portion 122, one tray 182 at the front position abutted against the guide member 123 is moved together with the guide member 123. As a result, one tray 182 on which the used tableware 180 is placed is arranged in the tableware pick-up tray arrangement portion 122.

When one tray 182 is moved from the receiving tray arranging portion 121 to the tableware picking tray arranging portion 122 and the guide member 123 is returned to the receiving tray arranging portion 121 by the robot arm 110, the receiving tray arranging portion 121 The remaining trays 182 in the tray 182 are shifted forward, the tray 182 in the foremost position is abutted against the guide member 123 to stop positioning, and the trays 182 in the subsequent trays 182 are also brought into contact with each other and stopped. ..

When the tray 182 is arranged in the tableware picking tray arrangement portion 122, the robot arm 110 points the hand 112 toward the tray 182 to hold the tableware 180, and the stereo camera 160 displays the image information of the tableware 180 on the tray 182. To be acquired. The image information of the tableware 180 acquired by the stereo camera 160 is the two-dimensional image information and / or the three-dimensional image information of the tableware 180, and when a plurality of tableware 180s are placed on the tray 182, the plurality of tableware 180 2D image information and / or 3D image information is acquired.
The image information of the tableware 180 acquired by the stereo camera 160 is transmitted to the control device 170, and the hand holding portion 180a of the tableware 180 held by the hand 112 is calculated by the tableware transport control device 171.

After that, the robot arm 110 sandwiches the hand holding portion 180a calculated by the tableware transport control unit 171 by the hand 112, picks up the tableware 180, and then prewashes the picked up tableware 180 in the prewashing unit 130. As a result, a part of the dirt adhering to the tableware 180 is removed before the dishwashing unit 150 cleans.

After that, the robot arm 110 accommodates the pre-washed tableware 180 in the washing rack 142 arranged in the rack arrangement unit 140.
Here, the robot arm 110 accommodates the tableware 180 at the accommodation position in the washing rack 142 calculated by the tableware transport control unit 171.
The tableware 180 picked up by the robot arm 110 is, for example, from the viewpoint of the efficiency of accommodating the plurality of tableware 180 in the washing rack 142 and the cleanability such as the degree of contact of the washing water inside the dishwashing unit 150. It is housed in the cleaning rack 142 according to a predetermined rule according to the shape of the above.
When the tableware transport control unit 171 determines that the cleaning rack 142 does not have an empty space for accommodating the tableware 180, the tableware transport control unit 171 sends the cleaning rack 142 to the dishwashing unit 150 by the rack delivery mechanism 143, and at the same time, the uppermost position is one step lower. Control is performed to move the cleaning rack 142 arranged in the above position to the uppermost position.

When the washing rack 142 is sent from the rack arrangement section 140 to the dishwashing section 150 by the rack delivery mechanism 143, the dishes 180 housed in the washing rack 142 are washed.
For the tableware 180 washed by the dishwashing unit 150, for example, the tableware 180 is taken out from the washing rack 142 by a robot arm (not shown) and stored as the washed tableware 180 in a storage container (not shown) or the like. The empty cleaning rack 142 is collected by a transfer robot (not shown) and automatically set in the rack arrangement unit 140.

[About image recognition]
In the image recognition of this embodiment, both two-dimensional information and three-dimensional information are used. The arrangement, shape, size, color, etc. of the tableware are identified by the two-dimensional image information, and the shape, edge height, color, etc. of the tableware are identified by the three-dimensional image information.
By inputting a two-dimensional input to the trained AI, information such as the shape, color, type, size, and arrangement of tableware is obtained as output. Then, three-dimensional information is added to the output of AI to more precisely specify the shape, gripping position, edge height, and color of the tableware.
Further, the present embodiment is not limited to machine learning using two-dimensional information, and machine learning using three-dimensional image information and image recognition of tableware by machine learning using both two-dimensional information and three-dimensional information. Is also possible.

The 2D image information is used to roughly identify the type, position, and size of the tableware, and the 3D image information refers to the information obtained from the 2D information to make the shape and grip position of the tableware more precise. It is used to identify to. In particular, the three-dimensional image information is used to obtain information on the height and angle of the gripping position in the tableware.

[About AI training]
First, data set generation will be described.
(1) Image preparation (1-1) Take 5 pictures of 11 kinds of tableware from different angles (1-2) Take 1 picture of 3 kinds of trays (1-3) Image of the side of the application place Take 5 pictures (1-4) Remove the background of 1-1 and 1-2 above and make them transparent.

(2) Image processing to increase the number of images (2-1) For trays ・ Brightness adjustment: γ = 0.7-1.5
・ Rotation: ± 10 °
・ Noise: Gaussian, Poisson, Salt and pepper (2-2) For tableware ・ Brightness adjustment: γ = 0.7-1.5
・ Rotation: ± 180 °
・ Affine transformation: 0 ° -45 °
・ Noise: Gaussian, Poisson, salt and pepper

(3) Conditions for placing tableware (3-1) The overlapping ratio with existing tableware must be less than 12.5% or 95% or more (3-2) Do not exceed the edge of the place where the tray can be placed before it rotates. (3-3) Do not exceed the edge of the tray before the tableware rotates (3-4) Do not exceed 50 retries

(4) Image composition (4-1) Randomly take an image of the side of one application place and use it as a background (4-2) Randomly take an image of one tray and use it as in (3) above. Increase the number of images Apply image processing and place them randomly within the range where the tray can be placed (4-3) Randomly take images of a minimum of 4 and a maximum of 8 transparent tableware, and the number of images as shown in (3) above. Place in random positions on the tray with image processing (4-4) Number of composites: 35,000
(4-5) Size of composite image: 1280x960 pixels

The software automatically generates the learning data for the photographed tableware by the following procedure.
-Take pictures of individual dishes and tableware trays.
-Multiple photos of tableware and tableware trays are combined by software to generate a pseudo tableware tray, and a label for learning is also automatically generated by the software.
-Perform machine learning based on the generated image / label data.

Next, AI training will be described.
(1) The framework is as follows.
a) OpenCV
b) Darknet
c) CUDA 10.2
(2) The YOLO v3-106 layer was used as the network.
(3) The parameters were set as follows.
a) Number of classes: 12 (tableware 11 and tray)
b) Epochs: 50,000
c) Batch_size: 8
d) Subdivisions: 4
e) Input_size: 608x352
f) Channels: 3
g) Learning_rate: 0.001
h) Momentum: 0.9
i) Decay: 0.0005
j) Anchors: Set as appropriate i), ii), iii)
k) Inflated parameters for each image when learning ・ Angle: 180
・ Saturation: 1.5
・ Exposure: 1.5
・ Hue: 0.1

The above settings are an example of the present embodiment, and the present embodiment is not limited to the above settings.

[About AI output results]
The output result of image recognition by AI is shown in FIG. Information such as the shape, color, type, size, and arrangement of tableware is accurately recognized by the AI of this embodiment. In FIG. 5, the tableware is recognized by the box, and appropriate labels are displayed according to the type of the tableware. Further, the AI of the present embodiment can recognize not only vessels such as plates, bowls, small bowls and cups, but also chopsticks, forks, spoon knives and cutlery. Furthermore, it is also possible to estimate irregular shapes such as other and garbage. For example, if paper scraps are recognized, they are removed by the arm.

The processing speed of image recognition by AI was about 25-34 fps, and the average recognition accuracy rate was confident: 92%.

[About picking up dishes from the washing rack]
When the robot arm picks up the tableware 180 from the washing rack 142, the tableware 180 is arranged on the washing rack 142 with the bottom surface visible. In this case, the tableware 180 can be picked up by the robot arm using only the two-dimensional image information acquired by the two-dimensional camera of the tableware 180.
In addition to the types of tableware 180 and the outer shape of the tableware 180, the thread bottom of the tableware 180, which is a characteristic area inside the outer shape, is multi-tasked in the image processing unit that processes the two-dimensional image provided in the control device (not shown) that controls the robot arm. By learning and recognizing, even the mask image of the thread bottom area can be clearly rounded.

For this purpose, a plurality of data including the image of the tableware 180, the area of the tableware 180, and the label are given to the image processing unit in advance, and further, a known shape as the shape of the feature area is given. In the example of the tableware 180, a plurality of sets of an image of the tableware 180, an area (contour) of the tableware 180, and a plurality of types (labels) of the tableware 180 are given for learning. Further, by comparing the thread bottom shape of a general thread bottom with a known shape (circle, square, rectangle, etc.), the position and posture can be calculated by the following method. Depending on the inclination of the thread bottom of the tableware, the shape such as a circle appears to be elongated and distorted on the two-dimensional image. Using this relationship, by calculating by calculation what a known shape, for example, a circle, looks like on a 2D image when captured by a 2D camera, the tableware is located in which direction as seen from the 2D camera. It is possible to estimate which way it is tilted.

Here, what is given to the image processing unit in advance is not so-called model data. Since the image processing unit is learned from the image information of the general tableware 180, it is possible to recognize various types, colors, and sizes of tableware that are not model data.

Generally, the position of the tableware output by image processing is inaccurate in the optical axis direction (depth direction) of the camera, and the distance in the depth direction is estimated if the dimensions of the object are not known in advance from the model data of the tableware 180. Can not. Since the image processing unit of this embodiment is modelless, it is difficult to accurately calculate the distance from the two-dimensional camera to the object.
Therefore, a method of estimating the position and posture of the tableware 180 without a model using the image processing unit and picking up the tableware 180 will be described below.

The orbit that brings the robot arm closer to the object is moved along the straight line connecting the origin of the camera coordinate system and the object so as to be along the optical axis direction of the image acquisition device, for example, a two-dimensional camera. Here, an example will be described in which the thread bottom of the tableware 180 is sucked and picked up by the suction surface of the suction bat provided at the tip of the robot arm. As a pickup trajectory in which the robot arm approaches the tableware along the optical axis direction of the two-dimensional camera, the robot arm is moved along a straight line connecting the origin of the camera coordinate system and the tableware 180.

When moving in a straight trajectory while maintaining the posture of the suction surface of the robot arm, the axis of the robot arm may rotate completely or pass through a peculiar posture depending on the position and posture, so that the robot arm follows the planned trajectory. It is necessary to avoid the inability to control the robot arm. For this reason, the operability can be secured at a certain value or more at the waypoint to the pick-up target end point posture according to the posture and position of the tableware 180, and the rotation amount of each axis of the robot arm is small around the tableware normal line. Use the swing angle and suction pad properly. That is, the robot arm can be provided with two suction pads having different orientations (for example, 90 degrees different orientations).

When the suction sensor provided in the vacuum path that evacuates the suction pad reacts, the tableware 180 is picked up by the robot arm as success, and then the tableware 180 is stored in a predetermined storage container or the like. Even when the depth distance is indefinite, the height of the mounting surface of the tableware 180 brings the suction pad of the robot arm closer to the thread bottom of the tableware 180 so that the tableware 180 can be picked up. If the suction pad of the robot arm reaches the height of the mounting surface of the tableware 180, it is judged that the robot arm has failed to be picked up, and the robot arm is pulled up to the standby position. The height of the mounting surface on which the tableware 180 is placed is a parameter that the robot arm driving device has grasped in advance. As a result, the image processing unit can estimate the position and posture of the tableware 180 and pick up the tableware 180 even in a state where the model image is not registered (modelless) by using the two-dimensional image.

If this image recognition device is applied to pick up the dishes 180 washed by the dishwashing section 150 from the washing rack 142, the dishes washed by the dishwashing section 150 can be picked up from the dishwashing section 150 or from the dishwashing section. The suction pad provided on the robot arm is brought into contact with the center of the thread bottom of the tableware 180 from the cleaning rack 142 put into the 150, the tableware 180 is picked up, and the tableware 180 is stored in the storage container for each type of the tableware 180. Is possible.

In the tableware automatic washing system 100 according to the first embodiment of the present invention, the tableware transport control unit 171 is arranged in the tray arrangement unit 120 based on the image information of the tableware 180 acquired by the stereo camera 160 as the tableware information acquisition unit. The hand holding portion 180a of the tableware 180 held by the hand 112 in order to pick up the tableware 180 on the tray 182 by the robot arm 110 is calculated, and the tableware 180 is held from the tray arrangement portion 120 while holding the hand holding portion 180a. Along with picking up, the robot arm 110 is controlled so that the picked up tableware 180 is housed in the washing rack 142.
Therefore, in the tableware automatic washing system 100 according to the first embodiment of the present invention, the tableware 180 can be picked up by holding the portion other than the bottom surface of the tableware 180 by the robot arm 110, and the tableware picked up by the robot arm 110. The 180 can be automatically housed in the cleaning rack 142.

Further, in the tableware automatic washing system 100 according to the first embodiment of the present invention, the stereo camera 160 can acquire information on the three-dimensional shape of the tableware 180 by the three-dimensional point group, and the tableware transport control unit 171 is a stereo camera. Since the hand holding portion 180a can be calculated by the 160 in consideration of the information on the height of the edge of the tableware 180, the robot arm 110 can hold the edge of the tableware 180 and pick up the tableware 180.

Further, in the tableware automatic cleaning system 100 according to the first embodiment of the present invention, the tray arranging unit 120 has a receiving tray arranging unit 121 for accepting a plurality of trays 182 on which used tableware 180 is placed and a receiving tray arranging unit 121. It has a tableware pick-up tray placement section 122 in which one tray 182 is transferred and arranged from the tray placement section 121 to a position where the tableware 180 is picked up by the robot arm 110, and the stereo camera 160 arranges the tableware pick-up tray. The image information of the tableware 180 on the tray 182 arranged in the unit 122 is acquired, and the tableware transport control unit 171 is based on the image information of the tableware 180 on the tray 182 arranged in the tableware picking tray arrangement unit 122. In order to calculate the hand holding portion 180a, one tray 182 on which the tableware 180 picked up by the robot arm 110 is placed can be reliably accommodated in the imaging region by the stereo camera 160.

Further, in the tableware automatic washing system 100 according to the first embodiment of the present invention, the stereo camera 160 acquires the image information in the storage part of the washing rack 142 arranged in the rack arrangement part 140, and the tableware transport control unit 171 obtains the image information in the storage part. , The empty space P of the cleaning rack 142 is calculated based on the image information in the accommodating portion of the cleaning rack 142 acquired by the stereo camera 160, and the tableware 180 picked up by the robot arm 110 for accommodating the cleaning rack 142. The storage position of the tableware 180 in the cleaning rack 142 is calculated from the image information of the above and the image information of the empty space P of the cleaning rack 142, and the tableware 180 is stored in the calculated storage position of the tableware 180 by the robot arm 110. Therefore, the tableware 180 can be efficiently stored in the washing rack 142.

In the tableware automatic washing method according to the first embodiment of the present invention, the tray arranging step of arranging the tray 182 on which the used tableware 180 is placed on the tray arranging unit 120 and the tableware 180 picked up by the robot arm 110 are washed. Acquired by a rack arrangement step of arranging the cleaning rack 142 to be accommodated for the purpose, a tableware information acquisition process of acquiring image information of the tableware 180 on the tray 182 arranged in the tray arrangement unit 120, and a tableware information acquisition process. Based on the image information of the tableware 180, a hand for calculating the hand holding portion 180a of the tableware 180 held by the hand 112 in order to pick up the tableware 180 on the tray 182 arranged in the tray arrangement unit 120 by the robot arm 110. Since the holding portion calculation step and the tableware transporting step of holding the hand holding portion 180a by the robot arm 110, picking up the tableware 180 from the tray arranging portion 120, and accommodating the picked up tableware 180 in the washing rack 142 are included. The tableware 180 can be picked up by holding the portion other than the bottom surface of the tableware 180 by the robot arm 110, and the tableware 180 picked up by the robot arm 110 can be automatically housed in the washing rack 142.

In the automatic tableware cleaning program according to the first embodiment of the present invention, since each step in the above automatic tableware cleaning method can be executed by the computer unit, the robot arm 110 holds a portion other than the bottom surface of the tableware 180. The tableware 180 can be picked up, and a control program for automatically accommodating the tableware 180 picked up by the robot arm 110 in the washing rack 142 can be executed by the computer unit.

Since the storage medium according to the first embodiment of the present invention stores the above-mentioned automatic dishwashing program, the automatic dishwashing program stored in the storage medium can be executed by the computer unit.

[Embodiment 2]
The tableware automatic washing system 200 according to the second embodiment of the present invention will be described with reference to FIG.
FIG. 7 is a block diagram showing the overall configuration of the tableware automatic washing system 200 according to the second embodiment of the present invention.

The tableware automatic washing system 200 according to the second embodiment is different from the tableware automatic washing system 100 of the first embodiment in that a plurality of stereo cameras 260 as the tableware information acquisition unit 260 are provided.
In addition, by assigning the corresponding 200-series reference numerals to the same components as the tableware automatic washing system 100 of the above-described first embodiment, the duplicated description will be omitted. Further, the parts not shown in FIG. 7 are designated by the same reference numerals as those in the first embodiment.

In the tableware automatic washing system 200, a stereo camera 260 that acquires two-dimensional image information or three-dimensional image information of the used tableware 180 placed on the tray 182 arranged in the tableware pick-up tray arrangement unit 222 is placed on the ceiling. It is fixed.
In the tableware automatic washing system 200, when the stereo camera 260 provided on the hand 212 of the robot arm 210 does not face the tray 182 arranged on the tableware picking tray arrangement unit 222, and the image information of the tableware 180 cannot be acquired. Even so, the image information of the tableware 180 can be acquired by the stereo camera 260 fixed to the ceiling.

Therefore, even when the stereo camera 260 fixed to the hand 112 is directed in a direction different from the tray 182 arranged in the tableware picking tray arrangement portion 222 with the movement of the robot arm 210. The stereo camera 260 fixed to the ceiling can continuously acquire the image information of the tableware 180 on the tray 182 arranged in the tableware pick-up tray arrangement unit 222.
Therefore, the tableware automatic washing system 200 according to the second embodiment of the present invention has the same effect as the tableware automatic washing system 100 according to the first embodiment, and the robot calculates the calculation speed of the hand holding portion 180a by the tableware transport control unit 271. It can be continuously executed regardless of the movement of the arm, and as a result, the robot arm 210 can be made faster.

Further, the stereo camera 260 may be further provided to acquire the image information of the tableware 180 on the tray 182 arranged in the receiving tray arranging unit 221.
By acquiring the image information of the tableware 180 acquired by the stereo camera 260, it is possible to acquire the image information of the tableware 180 on the tray 182 arranged in the tableware picking tray arrangement unit 222 in advance, and the tableware The calculation speed of the hand holding portion 180a by the transport control unit 271 can be increased.
Further, in order to further improve the storage efficiency and store the tableware 180 in the washing rack 142, not only the image information of the tableware 180 on the tray 182 arranged in the tableware picking tray arrangement portion 222 but also the tableware picking up next. The tableware transport control unit 271 may calculate the storage position of the tableware 180 in the washing rack 142 together with the image information of the tableware 180 on the tray 182 arranged in the tray arrangement unit 222.

In the automatic tableware washing systems 100 and 200 according to the first and second embodiments of the present invention, when the robot arms 110 and 210 pick up the tableware 180, the hands 112 and 212 hold the tableware 180. The tableware 180 may be sucked and picked up by the suction portion provided in.

Although some embodiments of the present invention have been described above, these embodiments exemplify an automatic dishwashing system for embodying the technical idea of the present invention, and specify the present invention to these. It is not a thing, and can be applied equally to those of other embodiments, and it is possible to omit, add, or change a part of these embodiments, or to combine the embodiments of each embodiment. be.

100, 200 ・ ・ ・ Automatic tableware washing system 110, 210 ・ ・ ・ Robot arm 110a, 210a ・ ・ ・ Robot control unit 111 ・ ・ ・ Arm 112 ・ ・ ・ Hand 113 ・ ・ ・ Holding piece 113a ・ ・ ・ Holding surface 120 , 220 ・ ・ ・ Tray arranging part 121,221 ・ ・ ・ Tray arranging part 121a for receiving ・ ・ ・ Tray mounting table 122, 222 ・ ・ ・ Tray arranging part 122a for tableware picking ・ ・ ・ Tray mounting table 122b, 222b・ ・ ・ Elevating drive unit 123, 223 ・ ・ ・ Guide member 130, 230 ・ ・ ・ Pre-washing unit 130a ・ ・ ・ Sinking table 140, 240 ・ ・ ・ Rack arrangement unit 141 ・ ・ ・ Elevating mechanism 141a, 241a ・ ・ ・ Elevating Drive unit 142 ・ ・ ・ Washing rack 143 ・ ・ ・ Rack delivery mechanism 143a ・ ・ ・ Drive unit 150, 250 ・ ・ ・ Dishwashing unit 150a, 250a ・ ・ ・ Dishwashing control unit 160, 260 ・ ・ ・ Stereo camera Tableware information acquisition department)
170, 270 ・ ・ ・ Control device 171,271 ・ ・ ・ Tableware transport control unit 180 ・ ・ ・ Tableware 180a ・ ・ ・ Hand holding part 182 ・ ・ ・ Tray

Claims (8)

A robot arm with a hand at the tip of the arm,
The tray placement section where the tray on which the tableware is placed is placed,
A rack arrangement part in which a washing rack for accommodating tableware picked up by the robot arm is arranged, and
A tableware information acquisition unit that acquires image information of tableware on the tray arranged in the tray arrangement unit, and
Based on the image information of the tableware acquired by the tableware information acquisition unit, the hand holding portion of the tableware held by the hand is recognized, the hand holding portion is held, and the tableware is picked up from the tray arrangement unit and picked up. A tableware transport control unit that controls the robot arm so that the tableware is housed in the washing rack.
An automatic dishwashing system characterized by being equipped with. The tableware information acquisition unit can acquire information on the two-dimensional and / or three-dimensional shape of the tableware.
The first aspect of claim 1, wherein the tableware transport control unit calculates the hand-holding portion of the tableware using information on the two-dimensional and / or three-dimensional shape of the tableware acquired by the tableware information acquisition unit. Dishware automatic cleaning system. The automatic tableware washing system according to claim 1 or 2, wherein the tableware information acquisition unit recognizes the shape of the tableware by machine learning. One tray is transferred from the receiving tray arranging part to a receiving tray arranging part that accepts a plurality of trays on which used tableware is placed and a position where the tableware is picked up by the robot arm from the receiving tray arranging part. It has a tray arrangement part for tableware collection, and
The tableware information acquisition unit acquires image information of the tableware on the tray arranged in the tableware pick-up tray arrangement unit.
Any of claims 1 to 3, wherein the tableware transport control unit calculates the hand holding portion based on image information of the tableware on the tray arranged in the tableware pick-up tray arrangement unit. The automatic dishwashing system according to item 1. The tableware information acquisition unit acquires image information in the storage unit of the washing rack arranged in the rack arrangement unit, and obtains image information.
The tableware transport control unit calculates an empty space of the washing rack based on the image information in the storage unit of the washing rack acquired by the tableware information acquisition unit, and uses the robot arm to store the empty space in the washing rack. From the image information of the picked up tableware and the image information of the empty space of the washing rack, the storage position of the tableware in the washing rack is calculated, and the tableware is stored in the calculated storage position of the tableware by the robot arm. The automatic dishwashing system according to any one of claims 1 to 4. The tray placement process in which the tray on which the used tableware is placed is placed in the tray placement section,
The rack placement process of arranging the washing rack that houses the dishes picked up by the robot arm for washing,
A tableware information acquisition process for acquiring image information of tableware on a tray arranged in the tray arrangement unit, and
Based on the image information of the tableware acquired by the tableware information acquisition step, the hand of the tableware held by the hand in order to pick up the tableware on the tray arranged in the tray arrangement portion by the robot arm. Hand holding part calculation process to calculate the holding part and
A tableware transporting process in which the hand holding portion is held by the robot arm, tableware is picked up from the tray arrangement portion, and the picked up tableware is stored in the washing rack.
An automatic dishwashing method characterized by including. A tableware automatic washing program, characterized in that each step in the tableware automatic washing method according to claim 6 can be executed by a computer unit. A storage medium that stores the automatic dishwashing program according to claim 7.

Images