JP7486986B2 - Control device and program - Google Patents

Description

An embodiment of the present invention relates to a control device and a program.

Conventionally, items such as luggage are handled (moved) using a moving device such as a robot arm. For example, a technology is known in which the location where the item to be moved is placed is photographed, the position, shape, and orientation of the item are recognized from the obtained image, and the item is picked up in order starting with the item with a clear surrounding area.

JP 2011-73066 A

However, in conventional technology, the order is determined based on the distance between the item to be grasped (moved) and other items around it, so it is not possible to take into account the relative positions of all the items, and there is room for further improvement in terms of efficiency.

The problem that this invention aims to solve is to provide a control device and program that can efficiently move items.

The control device of the embodiment includes a first connection unit, a second connection unit, and a control unit. The first connection unit is connected to a loading device that grasps an item placed on a loading section and moves the item. The second connection unit is connected to an imaging device that images the loading section from above. The control unit acquires an image captured by the imaging device via the second connection unit, executes a recognition process that recognizes the item placed on the loading section from the acquired image, calculates a distance between the items recognized in the recognition process, sets a priority for the order of movement for the items based on the distance, and controls the loading device via the first connection unit to move the items in an order according to the priority.

FIG. 1 is a diagram showing an example of the configuration of a cargo handling system according to the first embodiment. FIG. 2 is a diagram illustrating an example of a hardware configuration of the control device according to the first embodiment. FIG. 3 is a diagram illustrating an example of a functional configuration of the control device according to the first embodiment. FIG. 4 is a diagram for explaining the operation of the planning unit according to the first embodiment. FIG. 5 is a diagram for explaining the operation of the planning unit according to the first embodiment. FIG. 6 is a diagram for explaining the operation of the planning unit according to the first embodiment. FIG. 7 is a flowchart illustrating an example of a process executed by the control device of the first embodiment. FIG. 8 is a diagram for explaining an example of the operation of the control device according to the first embodiment. FIG. 9 is a diagram for explaining the operation of the planning unit according to the first modification of the first embodiment. FIG. 10 is a diagram illustrating an example of a functional configuration of a control device according to the second embodiment. FIG. 11 is a flowchart illustrating an example of a process executed by the control device according to the second embodiment.

The control device and loading/unloading system according to the embodiment will be described below with reference to the drawings.

First Embodiment
FIG. 1 is a diagram showing an example of the configuration of a cargo handling system according to the first embodiment.

As shown in FIG. 1, the loading and unloading system 1 includes a loading and unloading device 10, a camera 20, and a control device 30. Here, the loading and unloading device 10, the camera 20, and the control device 30 are connected via, for example, a serial cable or a LAN (Local Area Network), etc.

The loading device 10 is an example of a loading device. The loading device 10 is a device for loading (moving) an item W placed on the upper surface 2a of the pallet 2 to a destination position. The destination position of the item W may be any location to which the item W is to be moved, and may be, for example, a conveyor. The pallet 2 is an example of a placement section, and is placed, for example, on a horizontal floor surface G. The item W is, for example, a shaped or irregular three-dimensional object, and one or more items are placed on the upper surface 2a of the pallet 2. Note that, although an example in which the item W is placed on the pallet 2 is shown in this embodiment, this is not limiting, and the item W may be placed on, for example, the floor surface G. In this case, the area on the floor surface G on which the item W is placed corresponds to the placement section.

The loading and unloading device 10 comprises a base 111, an arm unit 112 supported by the base 111, and a gripping unit 113 provided at the tip of the arm unit 112. The base 111 is a housing fixed to a floor surface G. Hereinafter, the direction parallel to the floor surface G, i.e., the direction perpendicular to the direction in which gravity acts, is also referred to as the horizontal direction. In addition, the direction perpendicular to the floor surface G, i.e., the direction in which gravity acts and the opposite direction thereto, are also referred to as the up-down direction.

The arm unit 112 is, for example, a multi-joint robot arm. The arm unit 112 is configured to be capable of moving the item W held by the gripper unit 113 in the vertical direction and the horizontal direction.

For example, the arm portion 112 is composed of three movable portions 112a to 112c connected by multiple joints that can rotate around one or two axes. Movable portion 112a extends from the base 111. Movable portion 112b extends from movable portion 112a. Movable portion 112c extends from movable portion 112b.

The gripper 113 is attached to the tip of the arm 112 (movable part 112c). The gripper 113 pinches and grips the item W. Note that the gripping of the item W is not limited to the pinching gripping method. For example, instead of the gripper 113, a suction part that sucks and grips the item W by negative pressure may be used. Hereinafter, the action of gripping the item W by the gripper 113, or the action of gripping the item W and moving the item W to the target position by the arm 112 is also referred to as "picking". Note that the arm 112 and the gripper 113 are provided with a sensor device (e.g., a contact sensor, a weight sensor, etc.) for detecting the picking state of the item W.

The camera 20 is an example of an imaging device. The camera 20 is provided above the pallet 2, and captures an image of the top surface 2a of the pallet 2 from above. The camera 20 has an RGB camera 21 and a 3D camera 22. Note that both the RGB camera 21 and the 3D camera 22 include the pallet 2 in their imaging range (the range indicated by the dashed line in the figure).

The RGB camera 21 is an example of a first imaging device that captures a two-dimensional image. The RGB camera 21 is an imaging device that is capable of capturing color images. The RGB camera 21 outputs the two-dimensional image (hereinafter also referred to as an RGB image) obtained by imaging to the control device 30. This RGB image shows the state of the item W placed on the top surface 2a of the pallet 2 as viewed from above. Note that the two-dimensional image is not limited to an RGB image, and may be a grayscale image.

The three-dimensional camera 22 is an example of a second imaging device. The three-dimensional camera 22 is configured with a stereo camera, a 3D laser scanner device, or the like. The three-dimensional camera 22 captures (or measures) the top surface 2a of the pallet 2 from above to obtain three-dimensional information that three-dimensionally represents the height and shape of the item W placed on the top surface 2a of the pallet 2.

The three-dimensional camera 22 also generates a depth image and point cloud data from the acquired three-dimensional information and outputs them to the control device 30. Here, the depth image is an image obtained by converting distance information from the three-dimensional camera 22 to the item W placed on the top surface 2a of the pallet 2. The point cloud data is data that represents the shape of the surface of the item W placed on the top surface 2a of the pallet 2 as a three-dimensional point cloud. Hereinafter, the depth image and point cloud data are collectively referred to as a three-dimensional image.

The RGB camera 21 and the 3D camera 22 perform imaging in response to an imaging request from the control device 30, and transmit the RGB image and the 3D image obtained by the imaging to the control device 30. It is also possible to request imaging by specifying either the RGB camera 21 or the 3D camera 22, and in this case, the specified camera performs imaging individually.

In this embodiment, the three-dimensional image is generated by the three-dimensional camera 22, but the present invention is not limited to this, and the three-dimensional image may be generated by the control device 30. In this case, the three-dimensional camera 22 outputs three-dimensional information acquired by imaging to the control device 30, and the three-dimensional image is generated by the control device 30.

The control device 30 is an example of a control device in this embodiment. The control device 30 controls the operation of the loading and unloading device 10 based on the image capture results of the camera 20. Specifically, the control device 30 recognizes each of the items W placed on the upper surface 2a of the pallet 2 based on the RGB image and the 3D image input from the camera 20, and determines the picking order for each recognized item W. The control device 30 then controls the operation of the loading and unloading device 10 to move each of the items W placed on the upper surface 2a of the pallet 2 to the target position in the determined picking order.

In this embodiment, the control device 30 is separate from the loading device 10, but the loading device 10 and the control device 30 may be configured as an integrated unit. In this case, the loading device 10 and the control device 30 may be defined as the loading device. Also, the configuration including the loading device 10, the control device 30, and the camera 20 may be defined as the loading device.

Next, the hardware configuration of the control device 30 described above will be described. Figure 2 is a diagram showing an example of the hardware configuration of the control device 30.

As shown in FIG. 2, the control device 30 includes a processor 31, a ROM (Read Only Memory) 32, a RAM (Random Access Memory) 33, a camera interface 34, a loading device interface 35, and a memory unit 36.

The processor 31 is an example of a control unit and a computer. The processor 31 controls the overall operation of the control device 30. The processor 31 can be realized by a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), etc.

The ROM 32 stores various programs executed by the processor 31, setting information, etc. The RAM 33 is used as a work area for the processor 31.

The camera interface 34 is an example of a second connection unit. The camera interface 34 is a communication interface for connecting the camera 20. The camera interface 34 is communicatively connected to the camera 20, and controls communication between the device itself and the camera 20 under the control of the processor 31.

For example, the processor 31 transmits an image capture request to the camera 20 (RGB camera 21, 3D camera 22) via the camera interface 34. The processor 31 also acquires the RGB image and the 3D image transmitted from the camera 20 via the camera interface 34.

The loading/unloading device interface 35 is an example of a first connection unit. The loading/unloading device interface 35 is a communication interface for connecting the loading/unloading device 10. The loading/unloading device interface 35 is communicatively connected to the loading/unloading device 10, and controls communication between the loading/unloading device and the loading/unloading device 10 under the control of the processor 31.

For example, the processor 31 controls the operation of the loading device 10 via the loading device interface 35. The processor 31 also acquires sensing results indicating the picking status of the loading device 10 via the loading device interface 35.

The memory unit 36 is realized, for example, by a semiconductor memory element such as a flash memory or an auxiliary storage device such as a hard disk. The memory unit 36 stores various programs executed by the processor 31, setting information, and the like. The memory unit 36 also stores limit information indicating the vertical and horizontal movement limit positions of the loading device 10, the movable range of the loading device 10, the upper and lower limits of the dimensions of the item that the loading device 10 can grasp, and the like. Such limit information is used, for example, to derive a movement plan, which will be described later.

Next, the functional configuration of the control device 30 will be described with reference to FIG. 3. FIG. 3 is a diagram showing an example of the functional configuration of the control device 30.

As shown in FIG. 3, the control device 30 includes, as functional units, an acquisition unit 311, a recognition unit 312, a planning unit 313, and a plan execution unit 314. Some or all of these functional units may be software configurations that are realized by the processor 31 executing a program stored in the ROM 32 or the memory unit 36. In addition, some or all of the above-mentioned functional units may be hardware configurations that are realized by dedicated circuits provided in the processor 31. In this way, in this embodiment, the processor 31 functions as an example of a control unit.

The acquisition unit 311 acquires the RGB image and the 3D image captured by the camera 20 via the camera interface 34. Specifically, the acquisition unit 311 transmits an image capture request to each of the RGB camera 21 and the 3D camera 22 in response to an instruction from the recognition unit 312. Then, the acquisition unit 311 acquires the RGB image and the 3D image transmitted from each of the RGB camera 21 and the 3D camera 22.

When three-dimensional information is output from the three-dimensional camera 22, the acquisition unit 311 acquires the three-dimensional information acquired by the three-dimensional camera 22 via the camera interface 34.

The recognition unit 312 recognizes the position, shape, posture, etc. of the item W placed on the upper surface 2a of the pallet 2 based on the RGB image and the three-dimensional image. Specifically, the recognition unit 312 transmits a signal to the acquisition unit 311 to instruct the acquisition of an image in response to a recognition request from the planning unit 313. Then, the recognition unit 312 recognizes the position, shape, posture, etc. of each item W present on the upper surface 2a of the pallet 2 based on the RGB image and the three-dimensional image acquired by the acquisition unit 311.

The method for recognizing the items W is not particularly limited, and known techniques can be used. For example, the recognition unit 312 recognizes the position (horizontal position) and height (vertical height) of each item W present on the top surface 2a of the pallet 2 based on the RGB image and the depth image. The recognition unit 312 also recognizes the shape, orientation, etc. of each item W present on the top surface 2a of the pallet 2 based on the point cloud data.

When the acquisition unit 311 acquires three-dimensional information, the recognition unit 312 generates a depth image and point cloud data based on the three-dimensional information. The recognition unit 312 then recognizes each of the items W placed on the upper surface 2a of the pallet 2 using the generated depth image and point cloud data together with the RGB image.

The planning unit 313 derives a movement plan for moving each item W placed on the upper surface 2a of the pallet 2 to a destination position based on the recognition results of the recognition unit 312.

Specifically, the planning unit 313 instructs the recognition unit 312 to recognize the items W present on the top surface 2a of the pallet 2 by sending a signal to the acquisition unit 311 to instruct the acquisition unit 311 to acquire an image. The planning unit 313 calculates the distance between the items W recognized by the recognition unit 312, and sets priorities for the picking order for the items W based on the calculated distance. The planning unit 313 then derives a movement plan for moving the items W to the destination position in an order according to the priorities.

Here, an example of the operation of the planning unit 313 when setting a priority for item W will be described with reference to Figures 4 to 6.

Figures 4 to 6 are diagrams for explaining the operation of the planning unit 313, and show a schematic representation of the state of item W recognized by the recognition unit 312. In these figures, the numbers 1 to 4 at the end of item W refer to item numbers for identifying each item W assigned by recognition by the recognition unit 312. In the following, when identifying item W, it will be expressed as item Wn (n is the item number), which is a combination of item W and item number.

First, when the recognition unit 312 recognizes the items W present on the top surface 2a of the pallet 2, the planning unit 313 selects an item W to be the first item to be picked from among the recognized items W. Specifically, the planning unit 313 selects an item W that satisfies one or both of the following two conditions as the first item to be picked.

The first condition is that the item W is the one with the greatest height. The second condition is for the case where there are multiple items W, and that the item W is located at an end or corner of the group of items W. Here, the first condition takes into consideration the ease of picking the item W from above. The second condition is a condition for ensuring a larger distance between the items W to be picked in the process described below.

For example, if item W1, located at the top left corner, has the greatest height among items W1 to W4 shown in FIG. 4, the planning unit 313 sets the highest priority of 1 to item W1 so that it will be picked first. Note that in FIG. 4, item W1 is hatched with vertical lines to identify the item W for which a priority has been set (same below).

In the example of Figure 4, the planning unit 313 derives a movement plan that instructs item W1, which has a priority of 1, to be picked first, and item W1 is picked first under the control of the plan execution unit 314, which will be described later.

The planning unit 313 also calculates the distance between the item W1, which has been set to priority 1, and each of the other items W2 to W4 in order to select the next item W to be picked. Note that in FIG. 5, item W1 is hatched with diagonal lines in order to identify the item W that has already been picked (same below).

Specifically, as shown in FIG. 5, the planning unit 313 sets circles (hereinafter also referred to as circumscribing circles) C1 to C4 that circumscribe each of the items W1 to W4 based on the shapes of the items W1 to W4 recognized by the recognition unit 312. Here, a "circumscribing circle" means a circle that contains the item W and touches (circumscribes) at least two points on the outer edge of the item.

The planning unit 313 then uses the item W1, for which a priority was set in the previous process, as the base point and calculates the center-to-center distances d12, d13, and d14 from the center of its circumscribing circle C1 to the centers of the circumscribing circles C2 to C4 for each of the other items W2 to W4 as the separation distances. Hereinafter, the separation distances between the items W are also referred to as center-to-center distances.

The planning unit 313 also calculates radius sums r12 (= r1 + r2), r13 (= r1 + r3), and r14 (= r1 + r4) by adding the radius r1 of the circumscribing circle C1 for item W1 to the radii r2 to r4 of the other circumscribing circles C2 to C4.

The planning unit 313 compares the center-to-center distance and the sum of radii for each of items W2 to W4, and extracts items W whose center-to-center distance is greater than the sum of radii. For example, in the case of FIG. 5, the center-to-center distance d13 of item W3 is greater than the sum of radii r13, so the planning unit 313 extracts item W3 as a picking candidate. Similarly, the center-to-center distance d14 of item W4 is greater than the sum of radii r14, so the planning unit 313 extracts item W4 as a picking candidate. Similarly, the center-to-center distance d12 of item W2 is less than or equal to the sum of radii r12, so the planning unit 313 excludes this item W2 from the picking targets.

Here, item W2, whose center-to-center distance is equal to or less than the sum of the radii, may have been (or may be) affected by contact or other factors when picking item W1, which was used as the base point. In this case, even if item W2 is selected as the next picking target, there is a possibility that the position or posture of item W2 recognized in the recognition process may be deviated, and therefore picking may not be completed successfully. Therefore, by excluding item W2 from the picking targets, the operation of picking multiple items W in succession can be performed smoothly.

The exclusion setting for item W2 will be maintained until new recognition is performed by the recognition unit 312. Also, item W that has already been picked will be excluded from being picked.

The planning unit 313 then compares the center-to-center distances d13 and d14 between items W3 and W4 extracted as picking candidates, and selects one item W with a larger center-to-center distance, i.e., the item W with the largest center-to-center distance. In the example of FIG. 5, the center-to-center distance d14 for item W4 is greater than the center-to-center distance d13 for item W3, so the planning unit 313 selects item W4. The planning unit 313 then sets the next highest priority, 2, for the selected item W4. In this case, the planning unit 313 derives a movement plan that instructs the picking of item W2, which has been set to priority 2.

The planning unit 313 then sets the next priority of 3 for item W3 by performing the same process as described above, using item W4, which has been set to priority 2, as the base point. The planning unit 313 then derives a movement plan that instructs the picking of item W3, which has been set to priority 3.

Figure 6 shows the results of setting the priorities for items W1 to W4 as described in Figures 4 and 5. As described above, priorities 1, 3, and 2 are set for items W1, W3, and W4, respectively. In this case, as shown by the arrows in the figure, picking will be performed in order of priority, that is, item W1 → item W4 → item W3. Note that in Figure 6, an X is marked for item W2 to identify item W that has been excluded from the picking candidates.

When the planning unit 313 determines in the above process that there is no item W that is a candidate for picking, it outputs a recognition request to the recognition unit 312 to execute recognition processing based on new RGB images and 3D images captured by the camera 20. The planning unit 313 then sets the priority again based on the new recognition results acquired by the recognition unit 312.

In this way, the planning unit 313 sequentially sets the next highest priority to the other item W that is the greatest distance away from the item W for which a higher priority has been set. In other words, the planning unit 313 sets the other item W that is located farthest from the item W to be picked first as the next item to be picked. This makes it possible to reduce the impact on the item W to be picked next, even if the gripper 113 or the like comes into contact with another item W that is located in the vicinity of the item W when picking the item W. Therefore, the planning unit 313 can efficiently perform the operation of picking multiple items W in succession.

The circumscribing circle of the item W used by the planning unit 313 to calculate the separation distance defines the center of gravity and rotation range when the loading device 10 grasps the item W. Specifically, the center of gravity when the gripping unit 113 of the loading device 10 grasps the item W corresponds approximately to the center of the circumscribing circle of the item W. In addition, the loading device 10 may rotate the item W horizontally when picking the item W, and the rotation range corresponds approximately to the circumscribing circle of the item W. Therefore, by selecting and excluding picking candidates based on the center-to-center distance and the sum of radii of the circumscribing circles, the item W can be picked in an order that reduces the effects of contact and load collapse that occur during picking. In addition, even when picking an item W with an irregular shape such as item W4, the center of gravity and rotation range of the item W can be defined, so that the item W can be picked efficiently.

The planning unit 313 continues to use the recognition results obtained in one recognition process until it becomes impossible to extract picking candidates from the results, thereby deriving a movement plan for multiple items W. This allows the control device 30 to reduce the number of times that the camera 20 captures RGB images and 3D images, and the number of times that the recognition unit 312 executes the recognition process, thereby reducing the processing load and speeding up picking.

The planning unit 313 monitors the picking status of the item W notified by the loading and unloading device 10, and if an abnormality in the picking status is detected, similar to the above, it sends a recognition request to the recognition unit 312 to execute recognition processing based on a new RGB image and a 3D image. Here, an abnormality in the picking status may be, for example, a case where the scheduled item W cannot be picked, or a picking failure such as the item W falling off, etc.

Returning to FIG. 3, the plan execution unit 314 controls the operation (drive mechanism) of the loading/unloading device 10 based on the movement plan derived by the planner 313. Specifically, the plan execution unit 314 controls the operation of the loading/unloading device 10 based on the movement plan to grasp the item W to be picked that is placed on the top surface of the pallet 2 and move the item W to the destination position.

With the above-described functional configuration, the control device 30 can move the items W placed on the upper surface 2a of the pallet 2 to a target position using the loading and unloading device 10. More specifically, the control device 30 can move each of the items W placed on the upper surface 2a of the pallet 2 sequentially to a target position in the order of the set priority.

An example of the operation of the control device 30 described above will be described below. Here, FIG. 7 is a flowchart showing an example of the processing executed by the control device 30.

First, the planning unit 313 causes the acquisition unit 311 to acquire an RGB image and a three-dimensional image captured by the camera 20 by sending an image capturing request to the acquisition unit 311 via the recognition unit 312 (step S11). The recognition unit 312 executes a recognition process to recognize the item W based on the acquired RGB image and three-dimensional image (step S12).

Then, the planning unit 313 sets the index n, which specifies the priority value, to 1 (step S13). Next, the planning unit 313 determines whether or not an item W is present based on the recognition result of step S12 (step S14). More specifically, the planning unit 313 determines whether or not an item W other than the item W that has been picked is present in the recognition result.

If it is determined that an item W exists (step S14; Yes), the planning unit 313 determines whether the value of index n is 1 or not (step S15). Here, if it is determined that n=1 (step S15; Yes), the planning unit 313 selects one item W that satisfies either one or both of the first and second conditions described above from among the items W recognized in the recognition process of step S12 (step S16). Next, the planning unit 313 sets a priority n, i.e., a priority of 1, to the item W selected in step S16 (step S17), and then proceeds to step S22.

In the next step S22, the planning unit 313 derives a movement plan for the item W for which priority n has been set (step S22). Next, the plan execution unit 314 starts picking the item W for which priority n has been set by operating the loading and unloading device 10 based on the movement plan derived in step S22 (step S23).

The planning unit 313 monitors the picking status based on the sensing results transmitted from the loading and unloading device 10. When the planning unit 313 detects an abnormality in the picking status (step S24; Yes), it returns the process to step S11, thereby executing recognition processing based on the newly captured RGB image and 3D image.

If picking by the loading and unloading device 10 is completed normally (step S24; No), the planning unit 313 performs a deletion process to change the status of the item W included in the recognition result, which was assigned a priority n in the most recent process, to "picked" (step S25). Next, the planning unit 313 increments the value of the index n by 1 (step S26) and returns the process to step S14.

On the other hand, if it is determined in step S15 that n ≠ 1, that is, if the value of n is 2 or more (step S15; No), the planning unit 313 calculates the center-to-center distance and the sum of radii with respect to other items W, using the item W for which priority n-1 was set in the previous process as the base point (step S18).

Next, based on the calculation result of step S18, the planning unit 313 excludes from the picking candidates those items W whose center-to-center distance is equal to or less than the sum of radii (step S19). Next, the planning unit 313 determines whether or not there is an item W whose center-to-center distance is greater than the sum of radii among the remaining items W other than the items W that have already been picked (or have had their priority set) and the items W excluded in step S19 (step S20). Here, if it is determined that there is no item W (step S20; No), the planning unit 313 returns the process to step S11, thereby executing recognition processing based on newly captured RGB images and 3D images.

If it is determined that there is an item W whose center-to-center distance is greater than the sum of radii (step S20; Yes), the planning unit 313 sets a priority n to the item W whose center-to-center distance is the largest (step S21) and proceeds to step S22.

By the processing of steps S16 to S21 described above, a priority is sequentially set for each item W recognized in one recognition process. In addition, each time a priority is set for an item W, the movement plan for that item W is derived and the item W is picked, so that the items W are picked in order of priority.

Then, if the planning unit 313 determines in step S14 that item W is not present because all items W have been moved or for other reasons (step S14; Yes), the planning unit 313 ends this process.

Now, referring to FIG. 8, an example of the operation of the process in FIG. 7 described above will be described. Note that FIG. 8 is a diagram for explaining an example of the operation of the control device 30, and shows a schematic diagram of the state of the item W on the top surface 2a of the pallet 2 recognized by the recognition unit 312. Note that in FIG. 8, similar to FIGS. 4 and 5, the item W for which a new priority has been set is indicated by vertical hatching, and the item W for which picking has been completed, i.e., the item W11 for which the erasure process has been completed, is indicated by diagonal hatching.

As shown in FIG. 8(a), if item W is present in the recognition result of the recognition unit 312, the planning unit 313 selects the item W to be the first item to be picked based on the first and second conditions described above. For example, when the planning unit 313 selects item W11 at the upper left end (angle) from the 11 items W (W11 to W21) shown in FIG. 8(a), it sets the highest priority of 1 to this item W11. Item W11 with priority 1 set is picked by going through the processes of steps S22 and S23 described above.

The planning unit 313 continues to use the recognition results of the recognition unit 312 even after the picking of item W11 has been completed successfully, so the existence of item W11 remains in the recognition results. Therefore, as shown in FIG. 8(b), the planning unit 313 performs a deletion process to change the status of item W11 in the recognition results to "picked."

Next, as shown in FIG. 8(b), the planning unit 313 calculates the center-to-center distance and the sum of radii with respect to other items, using item W11, for which a priority was set in the previous process, as the base point. Here, if the center-to-center distance between item W11 and item W21 is the longest, the planning unit 313 sets the next highest priority of 2 for this item W21. Item W21, for which priority 2 has been set, is picked by going through the processes of steps S22 and S23 described above.

When picking of item W21 is completed normally, the planning unit 313 executes a deletion process for item W21 as shown in FIG. 8(c). Next, the planning unit 313 calculates the center-to-center distance and the sum of radii of the other items excluding item W11, using item W21, for which priority was set in the previous process, as the base point. If the center-to-center distance for item W20 is equal to or less than the sum of radii, the planning unit 313 excludes this item W20 from the picking targets. Also, if the center-to-center distance between item W21 and item W18 is the longest, the planning unit 313 sets the next priority of 3 for this item W18. Picking of item W18, for which priority 3 has been set, is executed by going through the processes of steps S22 and S23 described above.

When picking of item W18 is completed successfully, the planning unit 313 executes a deletion process for item W18, as shown in FIG. 8(d). Next, the planning unit 313 calculates the center-to-center distance and sum of radius of item W18, whose priority was set in the previous process, with respect to other items excluding items W11 and W21. Note that in FIG. 8(d), item W20, which was excluded from picking targets in the previous process (FIG. 8(c)), is represented by an x.

If the center-to-center distance between item W18 and item W14 is the maximum, the planning unit 313 sets the next highest priority of 4 for item W14. Item W14, which has been set to priority 4, is picked by going through the processes of steps S22 and S23 described above.

Then, the planning unit 313 repeats the same process for the remaining items W, and picks items W19 and W12 in sequence, as shown in Figures 8(e) and 8(f). In addition, items W that have been removed from the list of items to be picked are also accumulated through the process, and in the state of Figure 8(g), all of the remaining items W (W13, W15, W16, W17, W20) other than the items W that have been picked (W11, W12, W14, W18, W19, W21) have been removed from the list of items to be picked.

In this case, the planning unit 313 determines in the processing of step S20 that there are no objects whose center-to-center distance is greater than the sum of the radii, and returns the processing to step S11. As a result, new images are captured by the camera 20, and the recognition result of the new recognition processing performed by the recognition unit 312 includes the remaining objects W13, W15, W16, W17, and W20, as shown in FIG. 8(h).

Then, the planning unit 313 starts a new process of setting priorities for each of the items W13, W15, W16, W17, and W20 based on the recognition result in FIG. 8(h), so that picking can be performed in the order of priority.

As described above, the control device 30 executes a recognition process to recognize the items W placed on the pallet 2 from the images (RGB images, 3D images) captured by the camera 20, calculates the distance between the items W recognized in the recognition process, sets a priority for the items W in terms of the order of movement based on the calculated distance, and controls the loading and unloading device 10 to move the items W in an order according to the priority. In addition, when setting the above priority, the control device 30 sequentially sets the next highest priority to other items that are the greatest distance away from the item with the higher priority.

With the above configuration, the control device 30 can determine the picking order from the overall positional relationship of the items W placed on the top surface 2a of the pallet 2, so that a large distance can be ensured between the item W to be picked first and the item W to be picked next. This allows the control device 30 to prevent the effects of the previous picking from extending to the next item W to be picked, so that the operation of picking multiple items W in succession can be performed efficiently. Therefore, the control device 30 can efficiently pick items W.

The control device 30 can also set the priorities of multiple items in sequence based on the recognition results obtained from a single recognition process, and pick items in that order of priority. This allows the control device 30 to reduce the number of times the camera 20 captures RGB images and 3D images, and the number of times the recognition unit 312 executes the recognition process, thereby reducing the processing load and speeding up picking. The control device 30 can therefore efficiently pick items W.

The above-described embodiment can be modified as appropriate by changing a portion of the configuration or function of each of the above-described devices. Therefore, below, some modified examples of the above-described embodiment will be described as other embodiments. Note that below, differences from the above-described embodiment will be mainly described, and detailed descriptions of commonalities with the contents already described will be omitted. Also, the modified examples described below may be implemented individually or in appropriate combination.

(Variation 1)
In the above embodiment, a form was described in which the next picking target is another item W that has the largest center-to-center distance from the base item W, but the selection conditions for the picking target are not limited to this and further conditions may be added.

For example, after identifying another item W with the greatest center-to-center distance from the base item W, the identified other item W may be selected as a picking target only if the positional relationship between the other item W and items W present in its vicinity (hereinafter also referred to as surrounding items) satisfies a predetermined condition. The selection conditions of this modified example are described below with reference to FIG. 9.

Figure 9 is a diagram for explaining the selection conditions according to the first modified example. Here, each of the items W31, W32, and W33 shown in Figure 9 is an item W recognized by the recognition unit 312. In addition, the planning unit 313 calculates the center-to-center distances between item W31 and items W32 and W33, respectively, using item W31 as the base point, and has already identified item W33 with the longest center-to-center distance.

In the above-described embodiment, the planning unit 313 sets the identified item W33 as the next item to be picked by assigning it the next highest priority. However, in this case, the relationship between item W33 and the surrounding items around item W33 is not taken into consideration. Therefore, for example, as shown in FIG. 9, if item W32 is placed overlapping a portion of item W33, picking item W33 may result in contact with item W32 or cause item W32 to collapse.

Therefore, after identifying item W33, the planning unit 313 determines whether the positional relationship between item W33 and surrounding item W32 satisfies predetermined conditions, and selects item W33 as the next item to be picked only if it is confirmed that there is no risk of the load collapsing.

The judgment conditions can be set arbitrarily. For example, the judgment condition may be that the center-to-center distance between item W33 and item W32 is greater than the sum of their radii. Also, for example, the judgment condition may be that the height of the identified item W33 is greater than that of item W32. By adding such judgment conditions, an item W that is unlikely to come into contact with surrounding items or to collapse can be selected as the next item to be picked.

If the item W identified based on the center-to-center distance does not satisfy the above judgment conditions, the planning unit 313 excludes item W33 (and item W32) from the picking targets without setting the next priority to item W33 with the largest center-to-center distance from item W31. In this case, the planning unit 313 selects item W with the largest center-to-center distance from item W31 from the remaining items W (not shown) excluding items W33 and W32, and sets the next priority to item W.

(Variation 2)
In the above embodiment, a movement plan is derived for each priority setting, and picking is performed, but the present invention is not limited to this. For example, the planner 313 may first set the priority of each recognized item W, and then derive a movement plan at once for picking the items in the order according to the priority.

Specifically, the planning unit 313 selects a priority level of 1 for one item W by executing the processes of steps S16 and S17 described above, and then repeatedly executes the processes of steps S18 to S21 described above while incrementing the index n by 1. This allows the priority levels to be sequentially set for each of the recognized items W.

Then, when the planning unit 313 determines that there is no item whose center-to-center distance is greater than the sum of the radii (step S20; No), it derives a movement plan for picking items W in an order according to the priorities set up until that point. If an abnormality is detected during picking, the movement plan is discarded and the process returns to step S11.

This configuration can achieve the same effect as the above-described embodiment, and therefore item W can be picked efficiently in the same manner as the above-described embodiment.

Second Embodiment
Next, a second embodiment will be described. In this embodiment, a mode capable of detecting the occurrence of a collapse of cargo or the like by comparing the state of the item W before and after picking will be described. Note that the same reference numerals will be given to the same components as those in the first embodiment, and the description thereof will be omitted.

FIG. 10 is a diagram showing an example of the functional configuration of the control device 30a according to the second embodiment. As shown in FIG. 10, the control device 30a includes an acquisition unit 311, a recognition unit 312, a planning unit 313a, and a plan execution unit 314 as functional units.

The planning unit 313a is a functional unit corresponding to the planning unit 313 in the first embodiment, and has the same functions as the planning unit 313. In addition, as a unique function not included in the planning unit 313, the planning unit 313a requests the camera 20 to capture an RGB image via the recognition unit 312 when picking by the loading and unloading device 10 is completed. The planning unit 313a performs a difference check to detect the amount of change in the item W that occurred before and after picking, based on the RGB image newly acquired by the acquisition unit 311 and the previous RGB image used in recognition by the recognition unit 312.

Specifically, the planning unit 313a detects the amount of change in the area on the image where each of the items W exists between the previous RGB image and the new RGB image, and determines that there is a difference if the amount of change exceeds a threshold. The method of determining whether there is a difference is not particularly limited, and known techniques can be used. For example, the presence or absence of a difference may be determined by calculating the difference in pixel value for each pixel at the same pixel position in both the old and new RGB images, and comparing the average value of the calculated differences with a predetermined threshold. Also, for example, the presence or absence of a difference may be determined by using a machine learning model that is functionalized to output (infer) the presence or absence of a difference between the two images from the two images (image areas).

The planning unit 313a excludes from the difference check the items W for which the erasure process has been executed, that is, the items W for which picking has been completed, among the items W recognized by the recognition unit 312, and subjects the remaining items W to the difference check. Specifically, the planning unit 313a identifies items W for which picking has not been completed based on status information indicating whether or not the item has been picked, which is recorded in the recognition result, etc., and subjects the identified items W to the difference check. In this embodiment, a state with no differences is not limited to a state in which the new and old RGB images are completely identical, but is a concept that includes a state in which they are approximately equivalent while allowing for a certain degree of error.

If the difference check determines that there is a difference, it means that the placement position or posture of the item W has changed before and after picking. This means that the item W may have come into contact with other items W when picked, or that the item may have collapsed. In such a case, the movement plan derived based on the previous recognition result may not be able to capture the position of the item W, and therefore it may not be possible to pick the item correctly.

Therefore, if the planning unit 313a determines that there is a difference in the difference check, it requests the camera 20 to capture a three-dimensional image via the recognition unit 312. Next, the planning unit 313a causes the recognition unit 312 to execute recognition processing using the three-dimensional image newly acquired by the acquisition unit 311 and the previously acquired RGB image. Then, the planning unit 313a derives a movement plan for each of the recognized objects W based on the new recognition results.

An example of the operation of the control device 30a described above will be described below. FIG. 11 is a flowchart showing an example of the processing executed by the control device 30a. Note that the processing in steps S31 to S44 is similar to the processing in steps S11 to S24 in FIG. 7 described in the first embodiment, and therefore will not be described here.

In step S44, when picking is completed successfully (step S44; No), the planning unit 313a causes the acquisition unit 311 to acquire the RGB image captured by the RGB camera 21 by sending an RGB image capture request to the acquisition unit 311 via the recognition unit 312 (step S45).

Then, the planning unit 313a executes a difference check to compare the state of the item W before and after picking based on the RGB image acquired in step S45, the RGB image acquired in step S31 (the previous RGB image), and the recognition result in step S32 (the previous recognition result) (step S46). Next, the planning unit 313 determines whether or not there is a difference based on the result of the difference check (step S47).

If it is determined that there is no difference (step S47; No), the planning unit 313a executes the processes of steps S48 and S49, and then returns to step S34. Note that the processes of steps S48 and S49 are similar to steps S25 and S26 in FIG. 7 described in the first embodiment, and therefore will not be described here.

On the other hand, if it is determined that there is a difference (step S47; Yes), the planning unit 313a causes the acquisition unit 311 to acquire the 3D image captured by the 3D camera 22 by sending a request to capture a 3D image to the acquisition unit 311 via the recognition unit 312 (step S50). Next, the planning unit 313a returns the process to step S32, and causes the recognition unit 312 to execute recognition processing based on the RGB image acquired in step S45 and the 3D image acquired in step S50.

As described above, after picking of the item W is completed, the control device 30a performs a difference check to detect changes that occurred before and after picking based on the new RGB image captured after picking, the RGB image used in the recognition process, and the recognition result of the recognition process. Then, if the control device 30a determines that there is a change (a difference) as a result of the difference check, it performs a new recognition process using the new RGB image and 3D image, and sets a priority based on the recognition result of the recognition process.

As a result, even if the state of the item W placed on the upper surface 2a of the pallet 2 changes due to contact during picking, load collapse, or the like, the control device 30a can continue picking the item W based on the recognition result that reflects the changed state. Therefore, the control device 30 can efficiently pick the item W.

The above-described embodiment can be modified as appropriate by changing a portion of the configuration or function of each of the above-described devices. Therefore, below, some modified examples of the above-described embodiment will be described as other embodiments. Note that below, differences from the above-described embodiment will be mainly described, and detailed descriptions of commonalities with the contents already described will be omitted. Also, the modified examples described below may be implemented individually or in appropriate combination.

(Variation 1)
In the above embodiment, a request to capture a three-dimensional image is made on the condition that it is determined that there is a difference in the process of step S47 described in Fig. 11. However, the timing of acquiring a three-dimensional image is not limited to this, and the request to capture an image may be made at any timing in steps S44 to S47. As an example, the request to capture a three-dimensional image may be made at the same time as the request to capture an RGB image is made in step S45.

By performing the above process, if it is determined in step S47 that there is a difference, it is possible to move more quickly to recognition processing using a new RGB image and a 3D image. Note that if it is determined in step S47 that there is no difference, the acquired 3D image is discarded.

Although several embodiments (variations) of the present invention have been described above, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the gist of the invention. These embodiments and their variations are included in the scope and gist of the invention, and are included in the scope of the invention and its equivalents as set forth in the claims.

Reference Signs List 1 Cargo handling system 10 Cargo handling device 20 Camera 21 RGB camera 22 Three-dimensional camera 30 Control device 311 Acquisition unit 312 Recognition unit 313 Planning unit 314 Planning execution unit

Claims (10)

A first connection portion connected to a loading device that grasps an article placed on the placement portion and moves the article;
A second connection portion connected to an imaging device that images the placement portion from above;
a control unit that acquires an image captured by the imaging device via the second connection unit, executes a recognition process to recognize the items placed on the placement unit from the acquired image, calculates a distance between the items recognized in the recognition process, sets priorities for the items in a moving order based on the distance, and controls the loading and unloading device via the first connection unit to move the items in an order according to the priorities;
Equipped with
The control unit, in setting the priority, sequentially sets a next highest priority to another item that is the greatest distance away from the item for which a higher priority has been set. The control device according to claim 1 , wherein the control unit selects one of the recognized objects that has the greatest height, and sets the priority of the selected object to the highest. The control device according to claim 1 , wherein the control unit selects one of the recognized objects that is located at an end or corner of a group of objects, and sets the priority of the selected object to the highest level. The control device according to any one of claims 1 to 3, wherein the control unit sets a circumscribing circle that encompasses the object and touches at least two points on the outer edge of the object for each object recognized in the recognition process, and calculates the separation distance based on the distance between the centers of the circumscribing circles. The control device according to claim 4, wherein the control unit excludes another item from the priority setting targets when the distance between the item set with a higher priority and another item is less than or equal to the sum of the radii of the circumscribing circles for both items. The control device according to claim 5, wherein the control unit excludes another item from the priority setting targets when the distance between the other item that has the greatest distance from the item set with a higher priority and surrounding items present around the other item is less than or equal to the sum of radii. The control device according to any one of claims 1 to 6, wherein when the control unit detects an abnormality when the loading device grasps or moves the item, the control unit executes the recognition process using a new image captured by the imaging device. The control unit performs a difference check after the movement of the item is completed to detect the amount of change before and after the movement of the remaining items, excluding the moved item, based on a new image captured by the imaging device after the movement of the item and the previous image used in the recognition process, and if the amount of change exceeds a threshold, performs the recognition process using the new image. the second connection unit is connected to a first imaging device that captures a two-dimensional image and a second imaging device that captures a three-dimensional image;
The control device according to any one of claims 1 to 8, wherein the control unit recognizes the position, shape and posture of the item placed on the placement unit based on the two-dimensional image and the three-dimensional image captured by the first imaging device and the second imaging device. A computer of a control device including a first connection part connected to a loading device that grasps an article placed on the placement part and moves the article, and a second connection part connected to an imaging device that images the placement part from above,
an acquisition step of acquiring an image captured by the imaging device via the second connection part;
a recognition step of executing a recognition process for recognizing the article placed on the placement section from the image acquired in the acquisition step;
a calculation step of calculating a distance between the articles recognized in the recognition step;
a setting step of setting a priority for the item in a moving order based on the distance , and sequentially setting a second priority to another item that is the longest distance away from the item for which a higher priority has been set ;
a control step of moving the articles in an order according to the priority by controlling the loading device via the first connection unit;
A program for executing the above.

Images