JP6314017B2 - Product picking apparatus and program - Google Patents

Description

  The present invention relates to a commodity picking apparatus and a program.

  2. Description of the Related Art There is known a bulk picking device that acquires distance data and image data related to loosely stacked parts (workpieces) that are randomly stacked and recognizes the three-dimensional positions and orientations of a plurality of works (for example, Patent Documents). 1).

JP 2010-120141 A

  By the way, when a product is picked by a robot from a container in which a plurality of products are arranged, it is useful to accurately detect an arrangement mode of the plurality of products in the container. For example, even when the same product is arranged in the container with the same arrangement pattern, when a plurality of products in the container are close to one side, or when the products are in contact with each other, etc. This is because the exact arrangement position of a plurality of products may differ for each container.

  Therefore, in one aspect, an object of the present invention is to provide a product picking apparatus and a program that can accurately detect an arrangement mode of a plurality of products in a container.

According to one aspect of the present disclosure, when the bottom surface in the container is an XY plane, distance information in the Z-axis direction in the container in which a plurality of products to be picked by the robot are arranged, the plurality of positions in the X-axis direction and the Y-axis Sensors acquired at multiple positions in the axial direction;
Based on the distance change pattern along each of the X-axis direction and the Y-axis direction obtained from the distance information at the plurality of positions in the X-axis direction and the plurality of positions in the Y-axis direction, There is provided a product picking device including a processing device that calculates the number of products arranged in each of the Y-axis directions.

  A product picking apparatus and program capable of accurately detecting an arrangement mode of a plurality of products in a container are obtained.

The figure which shows roughly an example of the picking installation with which a goods picking apparatus is applied. The figure which shows schematically another example of the picking installation with which a goods picking apparatus is applied. The figure which shows an example of a structure of a goods picking apparatus. The figure which shows an example of a relationship between a distance sensor and the goods in a goods container roughly. The figure which shows an example of the hardware constitutions of a processing apparatus. The flowchart which shows an example of the initial process performed by a processing apparatus. The figure which shows the example of a distance change pattern. The figure which shows the other example of a distance change pattern. The figure which shows the other example of a distance change pattern. The figure which shows the other example of a distance change pattern. The figure which shows the other example of a distance change pattern. Explanatory drawing of the histogram of a horizontal direction, and the histogram of a vertical direction. 7 is a flowchart illustrating an example of a process (mount number calculation process) in step 604 in FIG. 6. Explanatory drawing of a mountain number calculation process. Explanatory drawing of a mountain number calculation process. 7 is a flowchart illustrating an example of processing (arrangement pattern specifying processing) in step 606 in FIG. 6. 7 is a flowchart showing another example of the process of step 604 in FIG. 6 (mountain number calculation process). Explanatory drawing of the calculation method of the average height of a division | segmentation point. 7 is a flowchart showing another example of the process (arrangement pattern specifying process) in step 606 in FIG. 6. Explanatory drawing of individual maximum height. 7 is a flowchart showing an example of processing (individual size detection processing) in step 608 in FIG. 6. Explanatory drawing of an example of the calculation method of maximum individual size. Explanatory drawing of an example of the calculation method of standard individual size. Explanatory drawing of another example of the calculation method of standard individual size. The flowchart which shows an example of the process which calculates the picking coordinate of each goods in the goods container C1. The flowchart which shows another example of the process which calculates the picking coordinate of each goods in the goods container C1. Explanatory drawing of an example of the provision rule of goods individual number. Explanatory drawing of the isolate | separated pixel set. Explanatory drawing of the division | segmentation method of a contact goods pixel set.

  Hereinafter, embodiments will be described in detail with reference to the accompanying drawings.

  FIG. 1 is a diagram schematically illustrating an example of a picking facility to which a commodity picking apparatus is applied.

  In the picking facility 1 shown in FIG. 1, the transport container C is sequentially transported along the product placement line 4. In addition, a conveyor etc. are used for this conveyance. A plurality of robots 50 are arranged along the product arrangement line 4. Each robot 50 picks the product P in the product container C 1 on the product container station 52 and places the picked product P in the transport container C on the product placement line 4. A new product container C1 is supplied to the product container station 52 every time all the products P in the product container C1 are picked and lost. Note that the transport container C is stopped at a predetermined position on the product placement line 4 at the time of product placement by each robot 50. The types of products P picked by each robot 50 may be different from each other. The product P to be picked is arbitrary, but may be, for example, a packaged product (for example, bread, confectionery, food). In this manner, in the picking facility 1 shown in FIG. 1, the transport container C is transported to the downstream side and delivered out while a product is placed by the predetermined robot 50 among the plurality of robots 50. It should be noted that the type and quantity of products to be placed in each transport container C (that is, which robot 50 arranges how many products) is determined by a store order or the like.

  FIG. 2 is a diagram schematically illustrating another example of the picking equipment to which the commodity picking apparatus is applied.

  In the picking facility 2 shown in FIG. 2, the transport container C is sequentially transported along the product placement line 4. In addition, a conveyor etc. are used for this conveyance. A plurality of robots 50 are arranged along the product arrangement line 4. Each robot 50 picks the product P in the product container C 1 on the product container station 52 and places the picked product P in the transport container C on the product placement line 4. A new product container C1 is supplied to the product container station 52 every time all the products P in the product container C1 are picked and lost. Note that the transport container C is stopped at a predetermined position on the product placement line 4 at the time of product placement by each robot 50. The types of products P picked by each robot 50 may be different from each other. The transport container C in which the product placement is completed is discharged to the product delivery line 6. Therefore, for example, the transport container C in which the product placement is completed by the most upstream robot 50 is discharged to the product delivery line 6 at that position. The transport container C discharged to the commodity delivery line 6 is transported downstream and discharged.

  In the picking facilities 1 and 2 shown in FIGS. 1 and 2, each robot 50 receives a picking instruction for picking coordinates from the processing apparatus 100 described later, and moves a product holding unit (arm) toward the picking coordinates to produce a product. Pick. Note that the application that gives a picking instruction to the robot 50 specifies, for example, a product to be picked by a product individual number and performs a picking instruction, and manages which product individual number product remains at the time after picking.

  The picking equipment 1 and 2 shown in FIGS. 1 and 2 are merely examples, and the product picking apparatus described below can be applied to picking equipment other than those shown in FIGS. 1 and 2. Also, the number of robots 50 is arbitrary, and ultimately the number of robots 50 may be one. The number of product container stations 52 may correspond to the number of robots 50.

  FIG. 3 is a diagram schematically illustrating an example of the configuration of the commodity picking apparatus.

  As shown in FIG. 3, the product picking apparatus 10 includes a distance sensor 70 and a processing apparatus 100. The distance sensor 70 is provided for each commodity container station 52 (see FIGS. 1 and 2). Below, the structure regarding one distance sensor 70 is demonstrated as a representative.

  The distance sensor 70 acquires information related to the distance in the predetermined product arrangement range of the product container C1 (hereinafter also referred to as “distance information”). The distance information is distance information in the Z-axis direction when the bottom surface of the product container C1 is the XY plane. Here, for the sake of convenience, the distance information represents the distance from the bottom surface of the product container C1 on the basis of the bottom surface of the product container C1 (the surface that contacts the lower surface of the product). However, the reference of the distance information may be other than the bottom surface of the product container C1. The distance information is acquired at a plurality of points along each of the X-axis direction and the Y-axis direction. The number of the plurality of points is an arbitrary number sufficient to generate a histogram to be described later, and the higher the number (resolution), the higher the accuracy of detecting the arrangement mode of each product P in the product container C1. . Hereinafter, as an example, the distance sensor 70 is assumed to be a sensor that generates a distance image, such as a Kinect sensor. The distance image includes distance information at a plurality of points along each of the X-axis direction and the Y-axis direction. Hereinafter, the X-axis direction is also referred to as “lateral direction”, the Y-axis direction is also referred to as “vertical direction”, and the Z-axis direction is also referred to as “height direction”. Therefore, the distance detected by the distance sensor 70 corresponds to the height from the bottom surface of the product container C1. When the product P is arranged in a predetermined product arrangement range of the product container C1, the distance information includes information related to the height of the product P.

  In the following, as a premise, each product P in the product container C1 is basically arranged in a two-dimensional manner in the X-axis direction and the Y-axis direction within a predetermined product placement range. That is, the products P are not stacked in a random manner in the product container C1. However, each product P in the product container C1 is partially overlapped in the Z-axis direction or contacted in the X-axis direction and / or the Y-axis direction due to the effects of how to put in the product, the inclination during transport, impact, etc. It can happen. Such a tendency is particularly likely to occur in a packaged product (for example, bread, confectionery, food). The predetermined product arrangement range is arbitrary, and may vary depending on the type of product. In the following, as an example, the predetermined product arrangement range corresponds to the entire bottom surface of the product container C1. The form of the product container C1 is arbitrary, and may be, for example, a box or a tray.

  FIG. 4 is a trihedral view schematically showing an example of the relationship between the distance sensor 70 and the product P in the product container C1. In the example shown in FIG. 4, a total of 6 products P are arranged in 3 rows horizontally by 2 vertically in the product container C1.

  The distance sensor 70 acquires distance information from above the product container C1, for example, as shown in FIG. As shown in FIG. 4, the distance sensor 70 may be disposed near the center of the commodity arrangement range of the commodity container C1. The position (height) in the Z direction of the distance sensor 70 may be set so that a distance image over the product arrangement range of the product container C1 can be acquired. Hereinafter, for convenience of explanation, the X-axis direction and the Y-axis direction are determined based on the horizontal and vertical directions of the product arrangement range of the product container C1, as shown in FIG. 4, and the corner of the product arrangement range of the product container C1 is defined as the origin. To do. When the size of the product arrangement range of the product container C1 is b0 and the length is a0, the distance sensor 70 acquires a distance image related to the product arrangement range (b0 × a0) of the product container C1.

  The processing apparatus 100 may be formed by a single processing apparatus or may include a plurality of processing apparatuses. For example, the processing apparatus 100 may separately include a processing apparatus that performs the following various processes (FIG. 6 and the like) and a processing apparatus that controls a picking facility such as the robot 50.

  FIG. 5 is a diagram illustrating an example of a hardware configuration of the processing apparatus 100.

  In the example illustrated in FIG. 5, the processing device 100 includes a control unit 101, a main storage unit 102, an auxiliary storage unit 103, a drive device 104, a network I / F unit 106, and an input unit 107.

  The control unit 101 is an arithmetic device that executes a program stored in the main storage unit 102 or the auxiliary storage unit 103, receives data from the input unit 107 or the storage device, calculates, processes, and outputs the data to the storage device or the like. To do.

  The main storage unit 102 is a ROM (Read Only Memory), a RAM (Random Access Memory), or the like, and a storage device that stores or temporarily stores programs and data such as an OS and application software that are basic software executed by the control unit 101. It is.

  The auxiliary storage unit 103 is an HDD (Hard Disk Drive) or the like, and is a storage device that stores data related to application software or the like.

  The drive device 104 reads the program from the recording medium 105, for example, a flexible disk, and installs it in the storage device.

  The recording medium 105 stores a predetermined program. The program stored in the recording medium 105 is installed in the processing device 100 via the drive device 104. The installed predetermined program can be executed by the processing apparatus 100.

  The network I / F unit 106 is an interface between the processing apparatus 100 and a peripheral device having a communication function connected via a network constructed by a data transmission path such as a wired and / or wireless line.

  The input unit 107 includes a keyboard having cursor keys, numeric input, various function keys, and the like, a mouse, a slice pad, and the like.

  In the example illustrated in FIG. 5, various processes described below can be realized by causing the processing apparatus 100 to execute a program. It is also possible to record the program on the recording medium 105 and cause the processing apparatus 100 to read the recording medium 105 on which the program is recorded, thereby realizing various processes described below. Note that various types of recording media can be used as the recording medium 105. For example, a recording medium for optically, electrically or magnetically recording information such as a CD-ROM, flexible disk, magneto-optical disk, etc., a semiconductor memory for electrically recording information such as ROM, flash memory, etc. It may be. Note that the recording medium 105 does not include a carrier wave.

  FIG. 6 is a flowchart illustrating an example of initial processing executed by the processing device 100. The process shown in FIG. 6 is a pre-process that is executed to obtain initial information related to the arrangement pattern of products in a certain product container (hereinafter also referred to as “model product container” for convenience of explanation). In the model product container, products are arranged in a predetermined arrangement pattern. In the case where the arrangement pattern varies depending on the type of factory equipment and product, a model product container may be prepared for each arrangement pattern. The process shown in FIG. 6 may be executed for each model product container having a different product arrangement pattern. For example, the process shown in FIG. 6 is executed for each product container station 52 that handles different types of products. At this time, when a new product container C1 having a different arrangement pattern is supplied to the corresponding product container station 52, the process shown in FIG. 6 may be executed using the product container C1 as a model product container. That is, the model product container may be the first product container supplied to the corresponding product container station 52. Here, as an example, processing for one model product container will be described as a representative. The same may be said about the process with respect to another model goods container.

  In step 600, the processing apparatus 100 acquires a distance image related to the model product container from the distance sensor 70. This distance image may be a distance image acquired by the distance sensor 70 when the model product container is located on the product container station 52.

  In step 602, the processing apparatus 100 generates a distance change pattern along the horizontal direction and a distance change pattern along the vertical direction based on the distance image related to the model product container.

  FIG. 7 is a diagram illustrating an example of the distance change pattern. FIG. 7A illustrates an example of the distance change pattern along the horizontal direction, and FIG. 7B illustrates the distance change pattern along the vertical direction. In FIG. 7, the distance change pattern along the X-axis direction at the position of Y = a of the model product container shown in FIG. 4 is shown by (A), and X = b of the model product container shown in FIG. A distance change pattern along the Y-axis direction at the position is indicated by (B).

  8A and 8B are diagrams showing another example of the distance change pattern. FIG. 8A shows an example of the distance change pattern along the horizontal direction, and FIG. 8B shows the distance change pattern along the vertical direction. . In the example shown in FIG. 8, a distance change pattern along the X-axis direction in which the heights at the respective positions in the Y-axis direction of the model product container shown in FIG. Further, in the example shown in FIG. 8, a distance change pattern along the Y-axis direction in which the heights at the respective positions in the X-axis direction of the model product container shown in FIG. 4 are summed is shown in (B). The distance change pattern shown in FIG. 8 is substantially the same as the histogram shown in FIGS.

  9 to 11 are diagrams showing other examples of the distance change pattern. FIG. 9A shows an example of the distance change pattern along the horizontal direction, and FIG. 9B shows the example of the distance change pattern along the vertical direction. The distance change pattern is shown. In the example shown in FIGS. 9 to 11, the distance change pattern is represented by a histogram. (A) is a horizontal histogram, and (B) is a vertical histogram. In the example shown in FIGS. 9 to 11, the arrangement pattern of the products in the model product container is different from the example shown in FIG. Specifically, in the example shown in FIG. 9, a histogram relating to five arrangement patterns in the horizontal direction and two in the vertical direction is shown. In the example shown in FIG. 10, a histogram relating to four arrangement patterns in the horizontal direction and four arrangement patterns in the vertical direction is shown. In the example shown in FIG. 11, a histogram relating to seven arrangement patterns in the horizontal direction and two arrangement patterns in the vertical direction is shown. In the following, m arrangement patterns in the horizontal direction and n arrangement patterns in the vertical direction are also referred to as “m × n arrangement patterns”.

  FIG. 12 is an explanatory diagram of a horizontal histogram and a vertical histogram. FIG. 12 schematically shows a distance image M related to a model product container in which products are arranged in a 4 × 4 arrangement pattern. The distance image M is provided with 15 × 13 grids (dotted lines), but these grids are for explanation. The horizontal histogram may be generated by averaging distance information in 13 squares in the vertical direction for each of the 15 columns in the horizontal direction, as indicated by symbol A. Similarly, as indicated by reference numeral B, the vertical histogram may be generated by averaging distance information in 15 squares in the horizontal direction for each of the 13 rows in the vertical direction. The distance information in one square may be an average of distance information in the square, a representative value, or arbitrary. In the example shown in FIG. 12, a histogram is generated with 15 × 13 squares, but the resolution for generating the histogram is arbitrary.

  Returning to FIG. 6, in step 604, based on the distance change pattern obtained in step 602, the processing apparatus 100 determines the number Nx of peaks in the change pattern in the X-axis direction and the peaks in the change pattern in the Y-axis direction. The number Ny is calculated. For example, in the example shown in FIG. 8, the number Nx of peaks in the X-axis direction (hereinafter also referred to as “X-axis direction peaks Nx”) is three, and the number Ny of peaks in the Y-axis direction (hereinafter “Y” The number of axial ridges Ny ”is two. Similarly, for example, in the example illustrated in FIG. 9, the X-axis direction mountain number Nx is five, and the Y-axis direction mountain number Ny is two. The peak of such a change pattern suggests the presence of a product, and the number of peaks indicates the number of products (the number of product placement). Note that the number of valleys may be calculated instead of the number of peaks.

  In step 606, the processing apparatus 100 specifies the arrangement pattern of the product in the model product container based on the X-axis direction mountain number Nx and the Y-axis direction mountain number Ny obtained in step 604. That is, the number of product arrangements in the X-axis direction and the Y-axis direction in the model product container is specified. For simplicity, the processing apparatus 100 may specify that the arrangement pattern is Nx × Ny. This is because the number of peaks in the change pattern in the X-axis direction (X-axis direction peak number Nx) represents the number of product arrangements in the X-axis direction, and the Y-axis direction peak number Ny represents the number of product arrangements in the Y-axis direction. This is because the possibility is high. However, when the arrangement pattern of Nx × Ny does not exist in a plurality of arrangement patterns that can actually be taken, other values may be used depending on the reliability of the X-axis direction mountain number Nx and the Y-axis direction mountain number Ny. The arrangement pattern may be specified in the manner described above. A plurality of arrangement patterns that can actually be taken may be registered in advance. This will be described later.

  In step 608, the processing apparatus 100 specifies the standard individual size of the product in the model product container based on the arrangement pattern specified in step 604. In the following description, “size” is assumed to be a two-dimensional size defined by respective lengths in the X-axis direction and the Y-axis direction. The standard individual size of the product may be registered in advance in association with the type of product or each arrangement pattern of the product. In this case, the processing apparatus 100 may adopt a standard individual size (registration information) corresponding to the arrangement pattern specified in step 604 above.

  According to the process shown in FIG. 6, it is possible to acquire information related to the arrangement pattern of products in the model product container. In addition, information on the standard individual size of the product can be acquired. Information regarding the arrangement pattern and standard individual size of the product acquired in this way is stored in a predetermined storage device (for example, the main storage unit 102) as an initial parameter. Hereinafter, information on the arrangement pattern of the product acquired in this way is also referred to as “arrangement pattern information”, and information on the standard individual size is also referred to as “individual size information”. The arrangement pattern information or the like can be used for calculating picking coordinates of each product in another product container C1 in which the product is arranged with the same arrangement pattern as the model product container.

  In the process shown in FIG. 6, the process in step 608 may be omitted in a configuration where the individual size information is not necessary for calculating picking coordinates described later (see, for example, FIG. 25).

  FIG. 13 is a flowchart illustrating an example of the process (mount number calculation process) in step 604 of FIG. The process shown in FIG. 13 is executed separately using distance change patterns in the respective directions in order to obtain the X-axis direction mountain number Nx and the Y-axis direction mountain number Ny, respectively. Hereinafter, as an example, a case where the X-axis direction mountain number Nx is calculated using a distance change pattern in the X-axis direction will be described. The same applies to the method of calculating the number of peaks in the Y-axis direction NY using the distance change pattern in the Y-axis direction.

  In step 1300, processing device 100 searches for a valley in the distance change pattern from the bottom side of the distance change pattern. In the example shown in FIG. 14, the distance change pattern is a histogram. As indicated by an arrow Z1 in FIG. 14, a valley is searched from the bottom side of the distance change pattern upward (the higher one). Specifically, the height h of a straight line S with a constant height (hereinafter referred to as “search line S”) is gradually increased from 0, and becomes lower than the search line S in the distance change pattern. When this range appears, the region is detected as a valley. In the example shown in FIG. 14, when the search straight line S rises to the height h1, five valleys V1 to V4 are detected.

  In step 1302, the processing apparatus 100 determines whether or not the horizontal width of each mountain portion is equal. The horizontal width of the peak is the width in the X-axis direction in this example, and corresponds to the width between adjacent valleys. The lateral width of the peak portion is calculated by the height of the search straight line S when the valley portion is detected in step 1300. For example, in the example shown in FIG. 14, since the valleys V1 to V4 are detected when the search straight line S rises to the height h1, the processing device 100 detects the lateral widths W1, W2 of the peaks at the height h1. , W3 is calculated.

  In this step 1302, equality does not necessarily mean complete equality, but is equal with a certain amount of error. For example, an error range within 30% may be made equal. For example, using the maximum value and the average value of the width of each peak, for each peak having a width of 50% or more of the maximum value and a width within an error range of 30% or less of the average value, You may consider that width is equal.

  If it is determined in step 1302 that the widths of the peaks are equal, the process proceeds to step 1304. If the widths of the peaks are not uniform, the process returns to step 1300. For example, in the example shown in FIG. 14, the horizontal widths W1, W2, and W3 of the peaks at the height h1 are not uniform (since W3 is significantly large), the process returns to step 1300. In this case, in step 1300, the height of the search line S is further increased until a new valley is found. As a result, in the example shown in FIG. 15, when the search straight line S rises to the height h2, a new valley V5 is detected. In step 1300, if the valley is not newly detected even when the height of the search straight line S is increased to the upper limit value, the number of peaks in the X-axis direction Nx is set to “indefinite” and the processing in FIG. 13 may be terminated. .

  In step 1304, the processing apparatus 100 calculates the number of peaks (in this example, the number of peaks in the X-axis direction Nx). For example, in the example illustrated in FIG. 15, the X-axis direction mountain number Nx is “4”.

  According to the processing shown in FIG. 13, the X-axis direction peak number Nx corresponding to the number of product arrangements in the X-axis direction can be accurately calculated by taking into account the lateral width of the peak part in the distance change pattern. Specifically, each mountain portion is basically formed due to each product in the model product container. Each product in the model product container is the same. Therefore, when each mountain part is detected corresponding to each product in the model product container, the width of each mountain part tends to be equal. According to the process shown in FIG. 13, by using such a tendency, it is possible to accurately calculate the number Nx in the X-axis direction corresponding to the number of product arrangements in the X-axis direction.

  In the process shown in FIG. 13, if the determination in step 1302 is affirmative, the number of peak portions at that time is adopted, but the height of the search straight line S is further increased, and the peak portions having a uniform lateral width. It may be confirmed whether or not the number increases. When the number of ridges having a uniform lateral width increases, the number of ridges (X-axis direction ridge number Nx) after the increase may be employed.

  FIG. 16 is a flowchart illustrating an example of the process (arrangement pattern specifying process) in step 606 of FIG. Here, it is assumed that all arrangement patterns that can actually be taken are registered in advance among arrangement patterns of products. Hereinafter, the arrangement pattern registered in this way is also referred to as a “registered arrangement pattern”.

  In step 1600, the processing apparatus 100 determines whether or not there is a registered arrangement pattern corresponding to the X-axis direction mountain number Nx and the Y-axis direction mountain number Ny obtained in step 604. That is, the processing apparatus 100 determines whether or not Nx × Ny registered arrangement patterns exist. If there is a registered arrangement pattern of Nx × Ny, the process proceeds to step 1602, and otherwise, the process proceeds to step 1604.

  In step 1602, the processing apparatus 100 determines the arrangement pattern of the model product container as an Nx × Ny arrangement pattern.

  In step 1604, the processing apparatus 100 calculates the height H of each peak from the bottom of the valley (hereinafter referred to as “peak height H”). The bottom of the valley corresponds to the current height of the search line S. The peak height H is calculated as a section (a range in the height direction) in which the horizontal width of each peak is equal. The peak height H corresponds to an increase in height when the height of the search straight line S is increased until the horizontal width of each peak is broken. In the example shown in FIG. 15, when the height of the search straight line S is increased to h3, the horizontal widths of the peaks are not uniform. Therefore, in this case, the peak height H is h3-h2. Note that the processing apparatus 100 calculates the peak height H for each distance change pattern in the X-axis direction and the Y-axis direction.

  In step 1606, the processing apparatus 100 selects a direction having a higher peak height H related to the distance change pattern among the X-axis direction and the Y-axis direction. For example, when the peak height H related to the distance change pattern in the X-axis direction is higher than the peak height H related to the distance change pattern in the Y-axis direction, the X-axis direction is selected.

  In step 1608, the processing apparatus 100 determines whether or not there is a registered arrangement pattern in which the number of mountains related to the direction selected in step 1606 is the number of products arranged in the same direction. For example, when the X-axis direction is selected, it is determined whether or not there is a registered arrangement pattern in which the X-axis direction mountain number Nx is the number of product arrangements in the X-axis direction. For example, when Nx = 4, for example, if there is a 4 × 4 registered arrangement pattern, the determination result of this step is affirmative. If the determination result in this step is affirmative, the process proceeds to step 1610. Otherwise, the process proceeds to step 1612.

  In step 1610, the processing apparatus 100 determines the registered arrangement pattern having the number of product arrangements in the same direction as the number of peaks in the direction selected in step 1606 as the model product container arrangement pattern. For example, when the X-axis direction is selected and the number of peaks in the X-axis direction Nx = 4, if a 4 × 4 arrangement pattern is registered, the 4 × 4 arrangement pattern is determined as the arrangement pattern of the model product container. .

  In step 1612, the processing apparatus 100 selects the direction that was not selected in step 1606. That is, the processing apparatus 100 selects a direction having a lower peak height H related to the distance change pattern from among the X-axis direction and the Y-axis direction.

  In step 1614, as in step 1608, the processing apparatus 100 determines whether there is a registered arrangement pattern in which the number of mountains in the direction selected in step 1612 is the number of product arrangements in the same direction. If the determination result in this step is affirmative, the process proceeds to step 1616. Otherwise, the process proceeds to step 1618.

  In step 1616, the processing apparatus 100 determines the registered arrangement pattern having the number of product arrangements in the same direction as the number of peaks in the direction selected in step 1612 as the arrangement pattern of the model product container.

  In step 1618, the processing apparatus 100 determines that the arrangement pattern of the model product container is “undefined”.

  According to the processing shown in FIG. 16, the arrangement pattern of the product in the model product container can be accurately identified by determining the consistency between the X-axis direction mountain number Nx and the Y-axis direction mountain number Ny and the registered arrangement pattern. (See step 1602).

  By the way, since each product in the model product container is two-dimensionally arranged as described above, the X-axis direction mountain number Nx and the Y-axis direction mountain number Ny calculated as described above based on the distance image are registered and arranged. It tends to be perfectly consistent with the pattern. However, in actuality, each product in the model product container partially overlaps in the Z-axis direction due to the way of putting in the product, the tilt or impact during transportation, the X-axis direction and / or the Y-axis. In some directions. In this case, the peak portion does not appear remarkably in the distance change pattern in the X-axis direction or the Y-axis direction, and as a result, the X-axis direction peak number Nx and the Y-axis direction peak number Ny may not completely match the registered arrangement pattern. It is possible.

  In this regard, according to the processing shown in FIG. 16, even if the X-axis direction mountain number Nx and the Y-axis direction mountain number Ny do not completely match the registered arrangement pattern, An arrangement pattern can be specified (see step 1610 and step 1616). At this time, the arrangement pattern of the product can be accurately identified by considering the number of peaks in the direction of the higher peak height H out of the number Nx in the X-axis direction and the number Ny in the Y-axis direction. . This is because the number of peaks in the direction with the higher peak height H of the number of peaks in the X-axis direction Nx and the number of peaks in the Y-axis direction Ny is more reliable than the number of peaks in the lower direction. It is because it is expensive.

  The process shown in FIG. 16 may be executed in combination with the process shown in FIG. Further, in the processing shown in FIG. 13, the X-axis direction mountain number Nx and the Y-axis direction mountain number Ny are calculated without considering the height of the mountain portion, but the height of the mountain portion is a certain value or more. Only in some cases, the X-axis direction mountain number Nx and the Y-axis direction mountain number Ny may be calculated.

  FIG. 17 is a flowchart showing another example of the process of step 604 in FIG. 6 (mountain number calculation process). Here, it is assumed that a registered arrangement pattern exists as a premise.

  In step 1700, for each number of product placements based on the registered placement pattern, the processing device 100 applies the breakpoints associated with each product placement number to the distance change pattern in the X-axis direction and the Y-axis direction, and sets the average height of the breakpoints. calculate. The break point corresponds to the position of the line that separates the products. For example, in the case of a 2 × 6 arrangement pattern, since the number of product arrangements in the X-axis direction is two, the break point in the X-axis direction is the width b0 (see FIG. 4) in the X-axis direction of the product arrangement range. Corresponds to half the position. In this case, since the number of product arrangements in the Y-axis direction is six, the break point in the Y-axis direction is the position of every 1/6 of the width a0 (see FIG. 4) in the Y-axis direction of the product arrangement range. Corresponding to

  FIG. 18 is an explanatory diagram of a method for calculating the average height of break points. In FIG. 18, the distance change pattern is shown as a curved pattern, but the distance change pattern may be a histogram. 18A shows a delimiter point L1 in the X-axis direction when the registered arrangement pattern is a 2 × 2 arrangement pattern. The dividing point L1 corresponds to the position of the dividing line when the X axis direction of the product arrangement range is equally divided into two. At this time, the average height of the separation point L1 corresponds to the height of the distance change pattern at the separation point L1 (the height of the intersection point P1 in FIG. 18A). Similarly, in FIG. 18, (B) shows delimitation points L2 and L3 in the X-axis direction when the registered arrangement pattern is a 3 × 4 arrangement pattern. Separation points L2 and L3 correspond to the positions of the dividing lines when the X-axis direction of the product arrangement range is equally divided into three. At this time, the average height of the separation points L2 and L3 corresponds to the average value of the heights of the distance change patterns at the separation points L2 and L3 (the heights of the intersections P2 and P3 in FIG. 18B). Similarly, in FIG. 18, (C) shows separation points L4 to L6 in the X-axis direction when the registered arrangement pattern is a 4 × 4 arrangement pattern. Separation points L4 to L6 correspond to the positions of the dividing lines when the X-axis direction of the product arrangement range is equally divided into four. At this time, the average height of the break points L4 to L6 corresponds to the average value of the height of the distance change pattern at each of the break points L4 to L6 (the height of the intersection points P4 to P6 in FIG. 18C). Similarly, in FIG. 18, (D) shows separation points L7 to L11 in the X-axis direction when the registered arrangement pattern is a 6 × 2 arrangement pattern. Separation points L7 to L11 correspond to the positions of the dividing lines when the X-axis direction of the product arrangement range is equally divided into six. At this time, the average height of the break points L7 to L11 corresponds to the average value of the height of the distance change pattern at each of the break points L7 to L11 (the height of the intersection points P7 to P11 in FIG. 18D).

  Returning to FIG. 17, in step 1702, the processing apparatus 100 selects a separation point with the minimum average height based on the average height of each separation point obtained in step 1700. At this time, when the average heights of two or more separation points are equal, the processing apparatus 100 selects the separation point having the larger number of product arrangements. For example, in the example illustrated in FIG. 18, the average height of the separation points related to the registered arrangement pattern of the 3 × 4 arrangement pattern is equal to the average height of the separation points related to the registered arrangement pattern of the 4 × 4 arrangement pattern. In this case, the separation points L4 to L6 having the larger number of product arrangements are selected. This is because the average height is low even if there are many break points (the number of product arrangements) because the break points are likely to be correct. That is, when the average height of two or more break points is the same, it is highly possible that the break point with the larger product placement number includes the break point with the smaller product placement number. This is because the break point with the larger number is likely to be correct.

  In step 1702, the processing apparatus 100 selects a break point with the minimum average height for each of the X-axis direction and the Y-axis direction.

  In step 1704, the processing apparatus 100 calculates the X-axis direction mountain number Nx and the Y-axis direction mountain number Ny based on the break point selected in step 1702. At this time, the X-axis direction peak number Nx is calculated by adding 1 to the number of separation points in the selected X-axis direction. Similarly, the number Ny in the Y-axis direction is calculated by adding 1 to the number of separation points in the selected Y-axis direction. For example, when the number of separation points in the selected X-axis direction is “3” (see FIG. 18C), the X-axis direction peak number Nx is 3 + 1 = 4.

  According to the processing shown in FIG. 17, by considering the consistency between the distance change pattern and the registered arrangement pattern, the X-axis direction mountain corresponding to the number of product arrangements (product arrangement pattern) in the X-axis direction and the Y-axis direction. The number Nx and the number Ny in the Y-axis direction can be calculated with high accuracy. Specifically, each mountain portion is basically formed due to each product in the model product container. When a break point exists between the peaks, the average height of the break points is reduced. Therefore, when each mountain portion is detected corresponding to each product in the model product container, the average height of the break points tends to be small. According to the processing shown in FIG. 13, by using such a tendency, the X-axis direction mountain number Nx and the Y-axis direction mountain number Ny corresponding to the number of product arrangements in the X-axis direction and the Y-axis direction can be calculated with high accuracy. Can do.

  FIG. 19 is a flowchart illustrating another example of the process (arrangement pattern specifying process) in step 606 of FIG. The processing shown in FIG. 19 is mainly different from the processing shown in FIG. In the following, processing that may be the same as the processing shown in FIG. 16 is assigned the same step number and description thereof is omitted. The process shown in FIG. 19 is executed in combination with the process shown in FIG. That is, the process shown in FIG. 19 is an example of the process of step 606 subsequent to step 604 of FIG. 6 when step 604 of FIG. 6 is realized by the process shown in FIG.

  In step 1904, the processing apparatus 100 calculates the individual maximum height H1 with respect to the average height of the breakpoints. As the average height of the breakpoints, the average height related to the breakpoints selected in step 1702 in FIG. 17 is used. The processing apparatus 100 calculates the individual maximum height H1 for each distance change pattern in the X-axis direction and the Y-axis direction. The individual maximum height H1 corresponds to the maximum height of the distance change pattern based on the average height of the break points. The maximum height of the distance change pattern is basically generated due to the product in the model product container. FIG. 20 is an explanatory diagram of the individual maximum height H1, in which (A) shows the case where the individual maximum height H1 is relatively large, and (B) shows the case where the individual maximum height H1 is relatively small. . FIG. 20A is an explanatory diagram of the individual maximum height H1 when the separation points L4 to L6 are selected in step 1702 of FIG. FIG. 20B is an explanatory diagram of the individual maximum height H1 when the delimiter points L7 to L11 are selected in step 1702 of FIG. Note that FIG. 20A and FIG. 20B have different distance change patterns.

  In step 1906, the processing apparatus 100 selects a direction having a higher individual maximum height H1 related to the distance change pattern among the X-axis direction and the Y-axis direction. For example, when the individual maximum height H1 related to the distance change pattern related to the X-axis direction is higher than the individual maximum height H1 related to the distance change pattern related to the Y-axis direction, the X-axis direction is selected.

  In step 1912, the processing apparatus 100 selects a direction having a lower individual maximum height H1 related to the distance change pattern from the X-axis direction and the Y-axis direction.

  According to the process shown in FIG. 19, as in the process shown in FIG. 16, the product in the model product container is determined by determining the consistency between the X-axis direction mountain number Nx and the Y-axis direction mountain number Ny and the registered arrangement pattern. Can be accurately identified (see step 1602).

  By the way, as described above, each product in the model product container partially overlaps in the Z-axis direction due to the effects of how to put in the product, tilt during transportation, impact, etc., or in the X-axis direction and / or the Y-axis. In some directions. In this case, the average height of the break points in the distance change pattern in the X-axis direction or the Y-axis direction is not significantly reduced (see FIG. 20B), and as a result, the number of peaks in the X-axis direction Nx and the peaks in the Y-axis direction It is possible that the number Ny does not completely match the registered arrangement pattern.

  In this regard, according to the process shown in FIG. 19, even when the X-axis direction mountain number Nx and the Y-axis direction mountain number Ny do not completely match the registered arrangement pattern, the registered arrangement pattern that partially matches is used. An arrangement pattern can be specified (steps 1610 and 1616). At this time, by considering the number of peaks in the direction with the highest individual maximum height H1 out of the number Nx in the X-axis direction and the number Ny in the Y-axis direction, the arrangement pattern of the product can be specified with high accuracy. . This is because the number of peaks in the direction with the highest individual maximum height H1 out of the number of peaks in the X-axis direction Nx and the number of peaks in the Y-axis direction Ny is more reliable than the number of peaks in the lower direction. It is because it is expensive.

  When the process shown in FIG. 19 and the process shown in FIG. 17 are combined, the number of peaks in the X-axis direction Nx and the Y-axis only in the case where the individual maximum height H1 is greater than or equal to a certain value in step 1704 shown in FIG. The number of direction peaks Ny may be calculated. That is, in the process shown in FIG. 17, the X-axis direction mountain number Nx and the Y-axis direction mountain number Ny are calculated without considering the individual maximum height H1, but the individual maximum height H1 is a certain value or more. Only in some cases, the X-axis direction mountain number Nx and the Y-axis direction mountain number Ny may be calculated.

  Further, when the process shown in FIG. 19 and the process shown in FIG. 17 are combined, in the above example, the consistency between the X-axis direction mountain number Nx and the Y-axis direction mountain number Ny and the registered arrangement pattern is determined. However, the consistency between the other parameters and the registered arrangement pattern may be determined. The other parameter is a parameter related to the number of peaks Nx in the X-axis direction and the number of peaks Ny in the Y-axis direction, and may be, for example, the number of separation points selected in Step 1702 of FIG.

  The process shown in FIG. 19 may be executed in combination with the process shown in FIG. For example, in the process shown in FIG. 16, when the process proceeds to step 1618, the process shown in FIG. 19 (the process shown in FIG. 17) may be executed. That is, when the product arrangement pattern cannot be specified by the method shown in FIG. 16, the product arrangement pattern may be specified by the method shown in FIG. Or, conversely, in the process shown in FIG. 19, when the process proceeds to step 1618, the process shown in FIG. 16 (the process shown in FIG. 13) may be executed.

  FIG. 21 is a flowchart showing an example of the processing (individual size detection processing) in step 608 of FIG.

  In step 2100, the processing apparatus 100 calculates the maximum individual size related to the product in the model product container based on the identified arrangement pattern. The maximum individual size may be calculated by dividing the size of the pixel area related to the product arrangement range by the number of products in the X-axis direction and the Y-axis direction based on the arrangement pattern. The example shown in FIG. 22 is an example in which a 4 × 3 arrangement pattern is specified. In this case, the maximum individual size is obtained by equally dividing the length in the X-axis direction of the pixel area related to the product arrangement range by 4 and It is calculated by equally dividing the length of the pixel area in the product arrangement range in the Y-axis direction by 3.

  In step 2102, based on the maximum individual size obtained in step 2100, the processing apparatus 100 determines whether or not a pixel set having a noise removal area that is greater than or equal to the maximum individual size exists in the distance image. The noise removal area is a set value for removing a pixel set related to noise in the distance image. A pixel set not less than the noise removal area and not more than the maximum individual size may be searched at a height (predetermined threshold) slightly higher than the bottom surface of the model product container. That is, a pixel set having a noise removal area that is equal to or greater than the maximum individual size is searched for from pixels having distance information that is equal to or greater than a predetermined threshold. This is because the cross section of the product (the cross section viewed in the Z-axis direction) is often a standard size at a height near the bottom surface of the model product container. In other words, when searching at a relatively high position from the bottom surface of the model product container, there is a high possibility that a pixel set smaller than the actual product size (a pixel set corresponding to the upper part of the product) is searched. If there is a pixel set in the distance image that is larger than the noise removal area and smaller than the maximum individual size, the process proceeds to step 2104; otherwise, the process proceeds to step 2106.

  In step 2104, the processing apparatus 100 calculates the size of the pixel set detected in step 2102 as the standard individual size related to the product in the model product container. The standard individual size may be represented by a circumscribed rectangle of the pixel set detected in step 2102 above. At this time, the circumscribed rectangle may be formed so as to be completely surrounded while circumscribing the pixels in the pixel set, or each side includes 50% or more of the pixels in the pixel set. It's okay. The standard individual size corresponds to a standard value that can be taken by the size of the pixel set of the product when one product in the model product container is separately detected from the distance image. For example, in the example shown in FIG. 23, two pixel sets (indicated by G1 and G2) having distance information equal to or greater than a predetermined threshold are detected. The pixel set G1 is a pixel set that is not less than the noise removal area and not more than the maximum individual size. Accordingly, in this case, the standard individual size is set to the size of the pixel set G1. The pixel set G2 corresponds to a pixel set when a plurality of products are detected without being separated. The pixel set G2 tends to occur when a plurality of products in the model product container are in contact with each other.

  In step 2106, the processing apparatus 100 calculates the standard individual size related to the product of the model product container by dividing the overall size of the pixel set having distance information equal to or greater than a predetermined threshold by the number of products based on the arrangement pattern. For example, in the example illustrated in FIG. 24, a pixel set G3 having distance information equal to or greater than a predetermined threshold is detected. In this case, the pixel set G3 can be regarded as a pixel set related to all commodities forming the arrangement pattern. The example shown in FIG. 24 is an example in which a 4 × 3 arrangement pattern is specified. In this case, the standard individual size is equal to 4 and 3 in the X-axis direction and the Y-axis direction, respectively. It is calculated by dividing into two.

  FIG. 25 is a flowchart illustrating an example of a process for calculating picking coordinates of each product in the product container C1. The process of FIG. 25 is a process related to the product container station 52 that handles the product container C1 containing the same product as the model product container described above. In the product container C1, products are arranged in the same predetermined arrangement pattern as the above-described model product container. Hereinafter, as an example, a process related to one product container station 52 that handles a product container C1 containing the same product as the above-described model product container will be described as a representative. The processing for the other merchandise container stations 52 may be similar.

  The process shown in FIG. 25 is performed on an arbitrary product container C1 supplied to the product container station 52 after the model product container described above.

  In step 2500, the processing apparatus 100 reads the arrangement pattern information (information relating to the arrangement pattern of the product in the model product container obtained by the process shown in FIG. 6).

  In step 2502, the processing apparatus 100 calculates picking coordinates of each product in the product container C1 based on the arrangement pattern information read in step 2500. For example, the processing apparatus 100 equally divides the product arrangement range of the product container C1 according to the arrangement pattern information. For example, when the arrangement pattern information represents a 2 × 6 arrangement pattern, the product arrangement range of the product container C1 is divided into 12 areas with a 2 × 6 pattern. The processing apparatus 100 calculates the center point of each area obtained by the division as the picking coordinates of each product.

  According to the process shown in FIG. 25, the picking coordinates of each product in the product container C1 can be calculated using the arrangement pattern information acquired as described above. This is particularly suitable when the arrangement pattern of products in the product container C1 supplied to the product container station 52 is not fixed. That is, it is suitable when the type of product in the product container on the product container station 52 changes. In this case, the process shown in FIG. 6 is executed using the first product container C1 after the type of the product in the product container on the product container station 52 is changed as a model product container, and then the process shown in FIG. What is necessary is just to calculate the picking coordinate of each product in the product container C1 after the change. The picking coordinates of each product in the model product container may be calculated in the same manner.

  Note that in the process shown in FIG. 25 (the same applies to the process shown in FIG. 26 described later), when arrangement pattern information is not obtained (see step 1618), fixed (default) picking coordinates may be adopted. When arrangement pattern information cannot be obtained, for example, when the upper part of the product is a square cross section and the products are in close contact with each other with no gap, or the products are in contact with each other in both the X-axis direction and the Y-axis direction. This can occur when For such products, there is a low possibility that the arrangement position in the product container C1 is shifted, or the shape is less likely to be reduced by shifting a plurality of products to one side, and picking by fixed picking coordinates is possible. There are many cases.

  FIG. 26 is a flowchart illustrating an example of processing for calculating the picking coordinates of each product in the product container C1. The process of FIG. 26 is a process related to the product container station 52 that handles the product container C1 containing the same product as the model product container described above. Hereinafter, as an example, a process related to one product container station 52 that handles a product container C1 containing the same product as the above-described model product container will be described as a representative. The processing for the other merchandise container stations 52 may be similar.

  In the following, processing while the type of product in the product container C1 on the product container station 52 (and the product arrangement pattern associated therewith) does not change will be described. When the type of the product in the product container C1 on the product container station 52 has changed, the process of FIG. 26 performs the process shown in FIG. 6 for the model product container after the change of the product type, similar to the process of FIG. It may be executed after being executed. The picking coordinates of each product in the model product container may also be calculated in the manner described below.

  In step 2600, the processing apparatus 100 reads the arrangement pattern information and the individual size information. The individual size information includes information on the standard individual size of the product in the model product container obtained by the processing of FIG. 6 (FIG. 21) as described above.

  In step 2601, the processing apparatus 100 acquires a distance image related to the product container C <b> 1 from the distance sensor 70.

  In step 2602, based on the distance image obtained in step 2600, the processing apparatus 100 creates a binary image in which the pixel whose height is equal to or greater than a predetermined threshold is 1 and the pixel whose height is less than the predetermined threshold is 0. create. The predetermined threshold value may correspond to the height of a position slightly higher than the bottom surface of the product container C1 (see the description of FIG. 21). Hereinafter, a set of pixels having a value of “1” is also referred to as a “product candidate pixel set”. A pixel set refers to a state in which any one pixel (a pixel having a value of “1”) in the set is adjacent to another pixel (a pixel having a value of “1”).

  In step 2603, the processing apparatus 100 assigns a product individual number based on the arrangement pattern information. The product individual number may be assigned in an arbitrary manner. For example, FIG. 27 shows an example of an individual number assignment rule for products (16 items in total) for a 4 × 4 array pattern. In FIG. 27, the product individual number is indicated by a number in a circle. In the example shown in FIG. 27, product individual numbers are assigned in ascending order in the direction from the top to the bottom in the Y-axis direction while proceeding from right to left along the X-axis direction.

  For example, the processing device 100 divides the outer shape surrounding all of the product candidate pixel sets by the arrangement pattern based on the arrangement pattern information, and assigns the product individual number in association with the center coordinates of each divided area. This divided area corresponds to an area where it is estimated that one product can exist, but the center coordinates here are not used as picking coordinates as described later. That is, the center coordinates correspond to unique information for specifying each product when a product individual number is assigned to each product in the product container C1. Note that the processing in step 2603 may be executed only during the first processing for a certain product container C1.

  In step 2604, the processing apparatus 100 calculates the maximum individual size based on the standard individual size of the individual size information. The processing apparatus 100 may calculate a value obtained by multiplying the standard individual size by a predetermined enlargement ratio K1 (> 1) as the maximum individual size. This is because a product candidate pixel set having a size larger than the standard individual size may correspond to one product due to the fact that the product is arranged obliquely in the product container C1. is there. Note that the processing apparatus 100 may use the maximum individual size of the individual size information as it is.

  In step 2606, the processing apparatus 100 calculates the minimum individual size based on the standard individual size of the individual size information. The processing apparatus 100 may calculate a value obtained by multiplying the standard individual size by a predetermined reduction rate K2 (<1) as the minimum individual size. This is because the product candidate pixel set having a size smaller than the standard individual size may correspond to one product because the product is arranged in one direction in the product container C1. Because there is.

  In step 2608, the processing apparatus 100 determines whether or not there is a candidate product pixel set having a size not less than the minimum individual size obtained in step 2606 and not more than the maximum individual size obtained in step 2604. For example, in the example shown in FIG. 28, two product candidate pixel sets G1 and G2 are detected. The product candidate pixel set G1 is a product candidate pixel set that is not less than the minimum individual size and not more than the maximum individual size. In this case, the determination result of this step 2608 is affirmative. In this step 2608, if there is a candidate product pixel set having a size not less than the minimum individual size and not more than the maximum individual size, the process proceeds to step 2610, and otherwise, the process proceeds to step 2612. A plurality of product candidate pixel sets having a size not less than the minimum individual size and not more than the maximum individual size may be detected.

  In step 2610, the processing apparatus 100 recognizes a product candidate pixel set having a size not less than the minimum individual size and not more than the maximum individual size as a pixel set related to one product (hereinafter referred to as “product pixel set”). In other words, the processing apparatus 100 recognizes a separated product pixel set related to one product. When a plurality of product candidate pixel sets having a size greater than the minimum individual size and less than or equal to the maximum individual size are detected, the processing apparatus 100 recognizes each product candidate pixel set as a product pixel set. Note that the processing apparatus 100 may calculate the size of the separated product pixel set, replace the calculated size with a standard individual size, or average the standard individual size of the individual size information and correct it.

  In step 2612, the processing apparatus 100 determines whether there is a product candidate pixel set larger than the maximum individual size obtained in step 2604. For example, in the example shown in FIG. 28, the commodity candidate pixel set G2 has a size larger than the maximum individual size. In this case, the determination result of this step 2612 is affirmative. If a product candidate pixel set larger than the maximum individual size exists, the process proceeds to step 2614. Otherwise, the process proceeds to step 2620. A plurality of product candidate pixel sets having a size larger than the maximum individual size may be detected.

  In step 2614, the processing apparatus 100 converts a product candidate pixel set having a size larger than the maximum individual size into a pixel set related to a plurality of products (products in contact with each other) (hereinafter also referred to as “contact product pixel set”). Recognize. The processing apparatus 100 calculates the number of divisions (the number of products included in the contact product pixel set) in the X-axis direction and the Y-axis direction with respect to the contact product pixel set. For example, the processing device 100 calculates an integer value obtained by dividing the length in the X-axis direction of the contact product pixel set by the length in the X-axis direction of the standard individual size in the X-axis direction included in the contact product pixel set. Calculated as the number of divisions. Similarly, the processing apparatus 100 determines the integer value obtained by dividing the length in the Y-axis direction of the contact product pixel set by the length in the Y-axis direction of the standard individual size in the Y-axis direction included in the contact product pixel set. It is calculated as the number of divisions. If the shape of the contact product pixel set is not rectangular, first, the contact product pixel set is equally divided by the length of the standard individual size in the Y-axis direction, and the X-axis direction of the standard individual size is obtained for each divided region. The number of divisions may be calculated by equally dividing the length. For example, in the example shown in FIG. 28, the processing device 100 equally divides the contact product pixel set G2 by the length of the standard individual size in the Y-axis direction, and the standard individual size for each of the divided regions G21, G22, G23. Are equally divided by the length in the X-axis direction, and the number of divisions is calculated.

  In step 2616, the processing apparatus 100 equally divides the contact product pixel set in each of the X axis direction and the Y axis direction based on the number of divisions in the X axis direction and the Y axis direction obtained in step 2614. When the shape of the contact product pixel set is not rectangular, the processing device 100 first equally divides the contact product pixel set in the Y-axis direction and equally divides each divided region in the X-axis direction. For example, in the example shown in FIG. 28, the processing device 100 equally divides the contact product pixel set G2 in the Y-axis direction, and equally divides each divided region G21, G22, G23 in the X-axis direction. As a result, the contact product pixel set is divided into a plurality of divided regions, and each divided region corresponds to one product. Hereinafter, each of the divided areas is referred to as a “divided frame”. The divided frame may be the divided area itself, or may be a rectangular frame that circumscribes the divided area. In the following, as an example, the division frame is assumed to be a rectangular frame circumscribing the division area. At this time, the divided frame may be arranged so as to completely surround the pixels in the corresponding divided area, or may be arranged so that each side includes, for example, 50% or more. Hereinafter, the pixel set in the division frame corresponds to a pixel set related to one product (each product pixel set in the contact product pixel set), and is also referred to as a product pixel set.

  In step 2618, the processing apparatus 100 corrects the size of the divided frame calculated as described above as necessary. For example, when the size of the divided frame is not less than the minimum individual size obtained in step 2606 and not more than the maximum individual size obtained in step 2604, the processing device 100 does not correct the size of the divided frame. On the other hand, if the size of the divided frame is less than the minimum individual size obtained in step 2606, the processing apparatus 100 corrects the size of the divided frame to the minimum individual size obtained in step 2606. If the size of the divided frame is larger than the maximum individual size obtained in step 2604, the processing apparatus 100 corrects the size of the divided frame to the maximum individual size obtained in step 2604.

  In Step 2618, processing device 100 corrects the position of a division frame as needed. This position correction may be necessary, for example, when the size of the divided frame obtained in step 2616 is corrected to the maximum individual size or the minimum individual size as described above. In other words, when the size of the divided frame is corrected as described above, there may be a case where a particular divided frame is excessive or insufficient in size. In addition, a gap may occur between the divided frames. In this case, the processing apparatus 100 is directed toward the divided frame with reference to the position of the divided frame related to the product with the earliest picking order (the product to be picked next) among the plurality of products related to the contact product pixel set. You may correct | amend the position of a division frame in the aspect which closes another division frame. For example, when the picking order corresponds to the ascending order of the product individual numbers shown in FIG. 27, the processing device 100 moves from left to right along the X-axis direction and from bottom to top along the Y-axis direction. The position of the dividing frame is reduced. Note that the processing device 100 may correct the divided frames related to the next picking target product in the same direction as necessary.

  In step 2620, the processing apparatus 100 calculates the center point of the area of each product pixel set as the picking coordinates related to each product. For example, in the case of a contact product pixel set, the processing device 100 calculates the center point of the division frame when the contact product pixel set is divided into each product pixel set as picking coordinates related to each product. In the case of a separated product pixel set (see step 2610), the processing device 100 may calculate the center point of the circumscribed rectangle of the separated product pixel set as picking coordinates. At this time, the circumscribed rectangle may be formed such that each side includes, for example, 50% or more of the pixels in the separated product pixel set.

  In step 2622, the processing apparatus 100 causes the robot 50 to pick the next product to be picked (may be one or more products) based on the picking coordinates calculated in step 2620. That is, the processing apparatus 100 outputs a picking instruction. As a result, the arrangement pattern of the merchandise in the merchandise container C1 changes, or all the merchandise in the merchandise container C1 disappears, so that a new merchandise container C1 is supplied to the merchandise container station 52. After outputting the picking instruction, the processing apparatus 100 updates the remaining state of the product in the product container C1 based on information from the robot 50. The remaining state of the product in the product container C1 is, for example, the number of remaining products (remaining number) or a product individual number related to the remaining products. Note that the information regarding the remaining state of the product in the product container C1 may be used, for example, in order to specify the next product to be picked in step 2618. When the process of step 2622 ends, the process returns to step 2601 and the process is repeated.

  According to the process shown in FIG. 26, the picking coordinates of each product in the product container C1 can be calculated based on the distance image related to the product container C1. As a result, it is possible to calculate the picking coordinates that accurately match the position of each product that can slightly change for each product container C1 due to the deviation of the product in the product container C1. In addition, since the arrangement pattern information is used, picking coordinates of these products can be calculated with high accuracy even when pixel sets relating to a plurality of products are not separated from each other in the distance image. In addition, according to the processing shown in FIG. 26, the size and position of the division frame that divides the contact product pixel set is corrected, so that the high accuracy that reflects the contact state (degree of closeness, etc.) of each product in the product container C1. The picking coordinates can be calculated.

  26, the processing from step 2601 is executed every time a new product container C1 is supplied to the product container station 52, and the product in the product container C1 is picked by the robot 50. It is executed whenever it is done. Thereby, it is possible to calculate the picking coordinates with high accuracy even for the position of each product that may be slightly different for each product container C1. Note that when the product in the product container C1 is picked by the robot 50, the position of the remaining product may slightly change due to disturbance caused by the picking operation. According to the process shown in FIG. 26, since the process from step 2601 is executed every time the product in the product container C1 is picked by the robot 50, the picking coordinates with high accuracy are calculated even when such a change occurs. can do.

  In the process shown in FIG. 26, the size of the divided frame is variable, but the size of the divided frame may be a fixed value. In this case, the fixed value may be the standard individual size of the individual size information or the standard individual size after the replacement described above.

  In the processing shown in FIG. 26, the center coordinates obtained in step 2603 are not used as picking coordinates. That is, the picking coordinates are calculated by the processing in steps 2604 to 2620. However, the center coordinates obtained in step 2603 may be used as picking coordinates. In this case, the processing of steps 2604 to 2620 may be omitted.

  By the way, as a measure for improving cost efficiency in logistics operations, product picking work using the robot 50 as described above can be considered. In general, the mainstream of robots is to target a product placed at a specific position as a picking object, or to target a product whose shape of the product itself is constant as one product type. For this reason, picking by a fixed coordinate determined in advance or picking by a robot is possible by using an image recognition technique to indicate the position of an object as robot coordinates.

  However, since the shape of the product in particular is not stable, in the stage where it is actually delivered for picking work (for example, to the product container station 52), each product is received in the product container C1 due to the influence of transport in the previous process. There are many cases where it is not arranged at a specific position. In recent years, due to diversification of consumer needs, the trend of delivery to stores from manufacturers / wholesalers has increased in small quantities, and the containers (transport containers C and product containers C1) delivered to stores are the same. However, there are various shapes for actual products. Further, due to the difference in the shape of the product, the arrangement pattern of the product in the product container C1 for each product type varies, but this information is not necessary for the logistics work by the operator. For this reason, although there is registration of the number in the product container C1, there is often no registration of the arrangement pattern. Or even if it is the same goods, the case where it becomes a multiple types of arrangement pattern by the characteristic (shipping factory) of a production line also occurs.

  Accordingly, when picking with a robot using fixed coordinates determined in advance for a bag product, picking mistakes frequently occur, which is not preferable in terms of work efficiency and quality. When picking coordinates of these product groups using image recognition technology, it is difficult to detect the location of each product using simple image recognition technology. This is due to the fact that the placement of each product is not at a specific position and that the separation of each product is unclear due to contact with the products in the product container C1.

  In this regard, according to the present embodiment, as described above, by performing the preprocessing as shown in FIG. 6, the product arrangement pattern can be detected even when the product arrangement pattern is not registered. Thereby, even if it is a case where the arrangement pattern of goods changes variously during operation, the arrangement pattern after change can be detected. In addition, according to the present embodiment, since the distance image is acquired for each product container C1 and the picking coordinates are calculated, accurate picking is possible even when the product placement position is slightly different for each product container C1. Coordinates can be calculated. In addition, since the distance image is acquired for each picking and the picking coordinates are updated, the picking coordinates with high accuracy can be calculated even when the arrangement position of the product is changed due to the picking. In addition, according to the present embodiment, the arrangement pattern of the product is specified in consideration of the consistency with the registered arrangement pattern. Therefore, even when the division of each product is unclear, accurate picking coordinates are calculated. be able to. Thereby, the frequency of picking mistakes can be reduced. Further, according to the present embodiment, as described above, since the arrangement pattern of the product is detected using the distance image, it is not necessary to use a combination of a plurality of sensors (for example, a camera and a laser sensor), and simple detection means can be used. Can be realized.

  Although each embodiment has been described in detail above, it is not limited to a specific embodiment, and various modifications and changes can be made within the scope described in the claims. It is also possible to combine all or a plurality of the components of the above-described embodiments.

  For example, in the above-described embodiment, in consideration of the fact that the product container following the model product container has the same product placement pattern as the model product container, the placement pattern is set only for the model product container using a histogram or the like. I have identified. However, in equipment or the like in which the arrangement pattern of the products in the product container changes variously, the arrangement pattern may be specified for each product container using the same histogram or the like.

In addition, the following additional remarks are disclosed regarding the above Example.
(Appendix 1)
Sensor that acquires distance information in the Z-axis direction in a container where a plurality of products to be picked by a robot are arranged at a plurality of positions in the X-axis direction and a plurality of positions in the Y-axis direction when the bottom surface in the container is the XY plane When,
Based on the distance change pattern along each of the X-axis direction and the Y-axis direction obtained from the distance information at the plurality of positions in the X-axis direction and the plurality of positions in the Y-axis direction, A merchandise picking apparatus including a processing apparatus that calculates the number of arrangements of the merchandise in each of the Y-axis directions.
(Appendix 2)
The distance change pattern along the X-axis direction includes a histogram obtained by averaging or accumulating the distance information at a plurality of positions in the Y-axis direction at a plurality of positions in the X-axis direction,
The product picking apparatus according to claim 1, wherein the distance change pattern along the Y-axis direction includes a histogram obtained by averaging or accumulating the distance information at the plurality of positions in the X-axis direction at the plurality of positions in the Y-axis direction.
(Appendix 3)
The processing device calculates the number of peaks in the distance change pattern along the X-axis direction, calculates the number of peaks in the distance change pattern along the Y-axis direction, and calculates the number of peaks The product picking device according to Supplementary Note 1 or 2, wherein the number of arrangements is calculated based on the above.
(Appendix 4)
The processing apparatus calculates the number of the crests in the distance change pattern along the X-axis direction and the number of the crests in the distance change pattern along the Y-axis direction, respectively. The product picking device according to attachment 3, wherein the number of the products is arranged in each direction.
(Appendix 5)
For the distance change pattern along the X-axis direction, the processing device calculates the number of peaks having the same width in the X-axis direction, and for the distance change pattern along the Y-axis direction, Y The product picking device according to appendix 3 or 4, which calculates the number of ridges each having an equal width in the axial direction.
(Appendix 6)
In the processing device, the number of the crests in the distance change pattern along the X-axis direction and the number of the crests in the distance change pattern along the Y-axis direction relate to the arrangement pattern of the products in the container. The product picking device according to any one of appendices 3 to 5, wherein the number of arrangements is calculated based on registration information to be matched when matching with registration information.
(Appendix 7)
The processing device calculates a height range in which the widths of the peaks in the distance change pattern along the X-axis direction are uniform, and the widths of the peaks in the distance change pattern along the Y-axis direction. Calculate a uniform height range,
The number of ridges in the distance change pattern along the X-axis direction and the number of ridges in the distance change pattern along the Y-axis direction do not match the registration information regarding the arrangement pattern of the product in the container. If only the number of peaks in the X axis direction and the Y axis direction in which the height range is wider matches the registration information, the processing apparatus registers matching information. The product picking device according to attachment 6, wherein the arrangement number is calculated based on the information.
(Appendix 8)
The product picking device according to appendix 1, wherein the processing device calculates an individual size of the product based on the calculated number of arrangements.
(Appendix 9)
The processing device, based on registration information related to the arrangement pattern of products in the container, for each registered arrangement pattern related to the registration information, for each of the distance change patterns along the X-axis direction and the Y-axis direction. Set the break point between them, and calculate the average value of the distance information at the set break point for each X-axis direction and Y-axis direction,
The processing device calculates the number of arrangements based on a separation point having the smallest average value of the distance information calculated in the X-axis direction and a separation point having the smallest average value of the distance information calculated in the Y-axis direction. The product picking device according to Supplementary Note 1 or 2, which is calculated.
(Appendix 10)
In the processing device, the number of arrangements in the corresponding direction is generated when a plurality of registered arrangement patterns have the same dividing point where the average value of the calculated distance information is the same in both the X-axis direction and the Y-axis direction. Item 10. The commodity picking apparatus according to appendix 9, wherein a delimiter point corresponding to the registered arrangement pattern having the largest number is used.
(Appendix 11)
In the processing apparatus, a registration point according to the registration information includes a separation point having the smallest average value of the distance information calculated in the X-axis direction and a separation point having the smallest average value of the distance information calculated in the Y-axis direction. The product picking device according to appendix 9 or 10, wherein the number of arrangements is calculated based on the registered arrangement pattern to be matched when matching with the arrangement pattern.
(Appendix 12)
The separation point with the smallest average value of the distance information calculated in the X-axis direction and the separation point with the smallest average value of the distance information calculated in the Y-axis direction are not consistent with the registered arrangement pattern related to the registration information. In this case, the processing device relates to the registration information corresponding to a separation point in a direction in which the difference between the average value of the distance information and the maximum value of the distance information is larger in the X-axis direction and the Y-axis direction. The product picking apparatus according to appendix 11, wherein the number of arrangements is calculated based on a registered arrangement pattern.
(Appendix 13)
The processing device calculates the picking coordinates for each product in another container in which the same product is arranged in the same arrangement pattern as the container based on the number of arrangements, any one of additional notes 1-12 The product picking apparatus according to claim 1.
(Appendix 14)
The sensor acquires the distance information regarding another container in which the same product is arranged in the same arrangement pattern as the container,
The processing apparatus calculates picking coordinates for each commodity in the another container based on the number of arrangements and the distance information regarding the another container. The product picking apparatus described.
(Appendix 15)
The processing device calculates an individual size of the product based on the number of arrangements, and calculates the picking coordinates based on the calculated individual size of the product and the distance information regarding the another container. Item 15. A product picking device according to appendix 14.
(Appendix 16)
16. The commodity picking apparatus according to appendix 14 or 15, wherein the sensor acquires the distance information each time a commodity in the another container is picked by the robot.
(Appendix 17)
The product picking device according to any one of appendices 14 to 16, wherein the processing device calculates the picking coordinates based on information related to a remaining state of a product in the another container.
(Appendix 18)
The sensor acquires a distance image including the distance information,
The processing device detects a pixel set having the distance information representing a predetermined distance or more with respect to a bottom surface of the another container in the distance image, and calculates the picking coordinates based on the detected pixel set. The commodity picking apparatus according to any one of Supplementary Notes 14 to 17.
(Appendix 19)
The processing device calculates the picking coordinates of the product related to the pixel set based on the center position of the pixel set when the size of the pixel set matches the individual size of the product. Goods picking equipment.
(Appendix 20)
When the size of the pixel set is significantly larger than the individual size of the product, the processing device determines a division mode of the pixel set based on the number of arrangements and the individual size of the product, The product picking apparatus according to appendix 18, wherein the picking coordinates of each product related to the pixel set are calculated based on a center position of each divided pixel set obtained by dividing the pixel set in the division mode.
(Appendix 21)
The program which makes the said processing apparatus perform the various processes by the said processing apparatus of the picking apparatus of any one of Additional remarks 1-20.
(Appendix 22)
Sensor that acquires distance information in the Z-axis direction in each container in the X-axis direction and each position in the Y-axis direction in the container where a plurality of products to be picked by the robot are arranged when the bottom surface in the container is the XY plane To obtain the distance information,
Based on the distance change pattern along each of the X-axis direction and the Y-axis direction obtained from the distance information at each position in the X-axis direction and each position in the Y-axis direction, the in-container X-axis direction and the Y-axis direction Calculating the number of products arranged in each direction of
A program that causes a computer to execute processing.
(Appendix 23)
Based on the number of arrangement of the product, calculating picking coordinates for each product in another container in which the same product is arranged in the same arrangement pattern as the container,
The program according to appendix 22, which causes a computer to execute processing.
(Appendix 24)
Sensor that acquires distance information in the Z-axis direction in a container where a plurality of products to be picked by a robot are arranged at a plurality of positions in the X-axis direction and a plurality of positions in the Y-axis direction when the bottom surface in the container is the XY plane To obtain the distance information,
Based on the distance change pattern along each of the X-axis direction and the Y-axis direction obtained from the distance information at the plurality of positions in the X-axis direction and the plurality of positions in the Y-axis direction, A product picking method, comprising: calculating the number of products arranged in each direction of the Y-axis direction.
(Appendix 25)
The said goods in additional notes 1-24 are bread | baked bread | baked confectionery and foodstuffs.

1, 2 Picking equipment 4 Product placement line 6 Product delivery line 10 Product picking device 50 Robot 52 Product container station 70 Distance sensor 100 Processing device

Claims (15)

Sensor that acquires distance information in the Z-axis direction in a container where a plurality of products to be picked by a robot are arranged at a plurality of positions in the X-axis direction and a plurality of positions in the Y-axis direction when the bottom surface in the container is the XY plane When,
Based on the distance change pattern along each of the X-axis direction and the Y-axis direction obtained from the distance information at the plurality of positions in the X-axis direction and the plurality of positions in the Y-axis direction, A merchandise picking apparatus including a processing apparatus that calculates the number of arrangements of the merchandise in each of the Y-axis directions. The distance change pattern along the X-axis direction includes a histogram obtained by averaging or accumulating the distance information at a plurality of positions in the Y-axis direction at a plurality of positions in the X-axis direction,
The product picking apparatus according to claim 1, wherein the distance change pattern along the Y-axis direction includes a histogram obtained by averaging or accumulating the distance information at the plurality of positions in the X-axis direction at the plurality of positions in the Y-axis direction. .   The processing device calculates the number of peaks in the distance change pattern along the X-axis direction, calculates the number of peaks in the distance change pattern along the Y-axis direction, and calculates the calculated peak The product picking apparatus according to claim 1, wherein the arrangement number is calculated based on a number.   In the processing device, the number of the crests in the distance change pattern along the X-axis direction and the number of the crests in the distance change pattern along the Y-axis direction relate to the arrangement pattern of the products in the container. The commodity picking apparatus according to claim 3, wherein the number of arrangements is calculated based on the matching registration information when matching with the registration information. The processing device calculates a height range in which the widths of the peaks in the distance change pattern along the X-axis direction are uniform, and the widths of the peaks in the distance change pattern along the Y-axis direction. Calculate a uniform height range,
The number of ridges in the distance change pattern along the X-axis direction and the number of ridges in the distance change pattern along the Y-axis direction do not match the registration information regarding the arrangement pattern of the product in the container. If only the number of peaks in the X axis direction and the Y axis direction in which the height range is wider matches the registration information, the processing apparatus registers matching information. The commodity picking apparatus according to claim 4, wherein the arrangement number is calculated based on information. The processing device, based on registration information related to the arrangement pattern of products in the container, for each registered arrangement pattern related to the registration information, for each of the distance change patterns along the X-axis direction and the Y-axis direction. Set the break point between them, and calculate the average value of the distance information at the set break point for each X-axis direction and Y-axis direction,
The processing device calculates the number of arrangements based on a separation point having the smallest average value of the distance information calculated in the X-axis direction and a separation point having the smallest average value of the distance information calculated in the Y-axis direction. The product picking device according to claim 1 or 2, wherein the product picking device is calculated. In the processing apparatus, a registration point according to the registration information includes a separation point having the smallest average value of the distance information calculated in the X-axis direction and a separation point having the smallest average value of the distance information calculated in the Y-axis direction. when consistent with the delimiting point arrangement pattern, based on the registration arrangement pattern having a delimiting point to be aligned, to calculate the number of placement, product picking device according to claim 6. A smallest-separated point average value of the distance information calculated in the X-axis direction, and the smallest-separated point average value of the distance information calculated in the Y-axis direction, separated points of the registration arrangement pattern according to the registration information If not aligned with, the processing unit, of the X-axis direction and the Y-axis direction, delimiting points corresponding to the average value and the distance towards direction delimiting point of a large difference between the maximum value of information of the distance information The product picking apparatus according to claim 7, wherein the number of arrangements is calculated based on a registered arrangement pattern related to the registration information having the information. The sensor acquires the distance information regarding another container in which the same product is arranged in the same arrangement pattern as the container,
The said processing apparatus calculates the picking coordinate with respect to each goods in the said another container based on the said arrangement number and the said distance information regarding the said another container, The any one of Claims 1-8. The product picking apparatus described in 1.   The processing device calculates an individual size of the product based on the number of arrangements, and calculates the picking coordinates based on the calculated individual size of the product and the distance information regarding the another container. The product picking apparatus according to claim 9.   The product picking device according to claim 9 or 10, wherein the sensor acquires the distance information every time a product in the another container is picked by the robot. The sensor acquires a distance image including the distance information,
The processing device detects a pixel set having the distance information representing a predetermined distance or more with respect to a bottom surface of the another container in the distance image, and calculates the picking coordinates based on the detected pixel set. The commodity picking apparatus according to any one of claims 9 to 11.   The processing device calculates the picking coordinates of the product related to the pixel set based on the center position of the pixel set when the size of the pixel set matches the individual size of the product. The product picking apparatus described.   When the size of the pixel set is significantly larger than the individual size of the product, the processing device determines a division mode of the pixel set based on the number of arrangements and the individual size of the product, The product picking apparatus according to claim 13, wherein the picking coordinates of each product related to the pixel set are calculated based on a center position of each divided pixel set obtained by dividing the pixel set in the division mode. Sensor that acquires distance information in the Z-axis direction in a container where a plurality of products to be picked by a robot are arranged at a plurality of positions in the X-axis direction and a plurality of positions in the Y-axis direction when the bottom surface in the container is the XY plane To obtain the distance information,
Based on the distance change pattern along each of the X-axis direction and the Y-axis direction obtained from the distance information at the plurality of positions in the X-axis direction and the plurality of positions in the Y-axis direction, Calculating the number of arrangement of the product in each direction of the Y-axis direction,
A program that causes a computer to execute processing.

Images