JP2022112068A - Loading operation support device, carrier device, loading operation support method, and loading operation support program - Google Patents

Abstract

To increase the robustness of arithmetic processing.SOLUTION: A forklift 1 comprises a photographing unit 24 that acquires a two-dimensional image and a three-dimensional image of a pallet 30 having a hole 32, and a control unit 27. The control unit 27 detects the hole 32 of the pallet 30 based on the acquired two-dimensional image and three-dimensional image.SELECTED DRAWING: Figure 7

Description

The present invention relates to a pick-up assistance device, a transport device, a pick-up assistance method, and a pick-up assistance program.

Conventionally, in a cargo handling operation using a forklift, the driver (operator) visually checks the holes (fork pockets) of the pallet and determines whether or not the forks can be inserted into the holes. As a result, the forks had to be inserted based on predictions, especially when the pallet was placed in a position that was difficult to see.

Therefore, for example, in the technique described in Patent Document 1, an image of a pallet is acquired by an imaging device (three-dimensional sensor) mounted on a forklift, and based on this image, the type of pallet and whether or not a fork can be inserted into its hole is determined. Judging.

Japanese Patent No. 6649796

However, the three-dimensional sensor may not be able to acquire data well, and in that case, processing will be hindered.
SUMMARY OF THE INVENTION Accordingly, it is an object of the present invention to improve the robustness of arithmetic processing.

A pickup support device according to the present invention includes:
2D imaging means for acquiring a two-dimensional image of a loading platform having holes;
3D imaging means for acquiring a three-dimensional image of the loading platform;
detection means for detecting the hole of the loading platform based on the two-dimensional image and the three-dimensional image;
Prepare.

A conveying device according to the present invention includes:
the pick-up support device;
and a holding portion that holds the cargo handling platform.

A pick-up support method according to the present invention includes:
the control means
a 2D imaging step of acquiring a two-dimensional image of a loading platform having holes;
a 3D imaging step of acquiring a 3D image of the loading platform;
a detection step of detecting the hole of the loading platform based on the two-dimensional image and the three-dimensional image;
to run.

A pick-up assistance program according to the present invention includes:
the computer,
2D imaging means for acquiring a two-dimensional image of a loading platform having holes;
3D imaging means for acquiring a 3D image of the loading platform;
detection means for detecting the hole of the loading platform based on the two-dimensional image and the three-dimensional image;
function as

ADVANTAGE OF THE INVENTION According to this invention, the robustness of arithmetic processing can be improved.

1 is a perspective view showing the appearance of a forklift according to an embodiment; FIG. 1 is a block diagram showing a schematic control configuration of a forklift according to an embodiment; FIG. 6 is a flowchart showing the flow of pallet position/orientation calculation processing according to the embodiment. FIG. 7 is a diagram for explaining pallet position/orientation calculation processing according to the embodiment; FIG. 7 is a diagram for explaining pallet position/orientation calculation processing according to the embodiment; FIG. 10 is a diagram showing a display example of a result of the pallet position/orientation calculation process according to the embodiment; 10 is a flowchart showing the flow of pallet position/orientation calculation processing according to a modification of the embodiment. FIG. 11 is a diagram for explaining pallet position/orientation calculation processing according to a modification of the embodiment; FIG. 11 is a diagram showing an example of an input screen for pallet size information according to a modification of the embodiment;

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[Forklift configuration]
FIG. 1 is a perspective view showing the appearance of a forklift 1 according to this embodiment.
A forklift 1 according to the present embodiment is an example of a carrier apparatus according to the present invention, and includes a vehicle body 11 , forks 12 , an elevating body (lift) 13 , a mast 14 and wheels 15 . The mast 14 is provided in front of the vehicle body 11 and is driven by a drive source (not shown) to tilt forward and backward relative to the vehicle body 11 . The elevating body 13 is driven by a drive source (not shown) and ascends and descends along the mast 14 . A pair of forks 12 are attached to the lifting body 13 as a holding portion for holding a load, a pallet 30, and the like.
The pallet 30 is an example of a cargo handling table according to the present invention, and as shown in FIG. The pallet 30 is formed in the shape of a short rectangular plate and has two holes (fork pockets) 32 in each side surface 31 into which the pair of forks 12 are inserted. Each hole 32 extends along the four sides of the pallet 30 and is defined by columnar portions 33 located on the front, back, left and right sides thereof.

FIG. 2 is a block diagram showing a schematic control configuration of the forklift 1. As shown in FIG.
As shown in this figure, the forklift 1 includes a drive unit 21, an operation unit 22, a display unit 23, an imaging unit 24, a storage unit 26, and a control unit 27 in addition to the above configuration. The pick-up support device according to the present invention includes at least a photographing section 24 and a control section 27 .

The drive unit 21 includes a travel motor, a steering motor, and a cargo handling motor (all of which are not shown), which are various drive sources of the forklift 1 . The travel motor drives drive wheels of the wheels 15 . The steering motor rotates (steers) the steered wheels of the wheels 15 . The cargo-handling motor is a driving source for raising and lowering the lifting body 13 and tilting the mast 14 .

The operation unit 22 is operation means for the driver to perform various operations. The operation unit 22 includes, for example, a handle, a pedal, a lever, various buttons, and the like, and outputs an operation signal to the control unit 27 according to the content of these operations.
The display unit 23 is, for example, a liquid crystal display, an organic electroluminescence display, or other displays, and displays various information based on display signals input from the control unit 27 . Note that the display unit 23 may be a touch panel that also serves as a part of the operation unit 22 .

The photographing unit 24 is capable of obtaining an image including depth (distance) information, and in this embodiment is an RGB-D camera (depth camera) capable of obtaining an image including three-dimensional point cloud data. That is, the photographing unit 24 can acquire three-dimensional information and an RGB image (two-dimensional image). The photographing unit 24 is attached to, for example, the lifting body 13 or the mast 14 facing forward, and photographs the front of the forklift 1 .
Note that the imaging unit 24 may acquire a two-dimensional image of an imaging target and a three-dimensional image including depth information. That is, the photographing unit 24 may separately include means for obtaining a two-dimensional image of the object to be photographed and means for obtaining a three-dimensional image. Also, the photographing unit 24 may be an infrared camera capable of acquiring an infrared image instead of an RGB image, for example. In this case, an infrared image can be used, so it can be used even in the case of insufficient illumination.

The storage unit 26 is a memory configured by RAM (Random Access Memory), ROM (Read Only Memory), etc., stores various programs and data, and also functions as a work area for the control unit 27 . CAD data (design data) of the pallet 30 is stored in advance as the shape information of the pallet 30 in the storage unit 26 of the present embodiment.
The control section 27 controls the operation of each section of the forklift 1 . Specifically, the control unit 27 operates the driving unit 21 based on the operation content of the operation unit 22, develops a program stored in advance in the storage unit 26, and cooperates with the developed program to perform various operations. perform processing.

[Forklift operation]
Next, a pickup operation when the forklift 1 inserts the forks 12 into the pallet 30 will be described.
In this pick-up operation, the position and attitude of the pallet 30 are calculated based on the image of the pallet 30, and the insertion of the forks 12 into the holes 32 of the pallet 30 is favorably assisted based on the results.
FIG. 3 is a flow chart showing the flow of the position/orientation calculation processing of the pallet 30, FIGS. 4 and 5 are diagrams for explaining this processing, and FIG. 6 shows an example of the result display in this processing. It is a diagram.
This process is executed by the control unit 27 reading and expanding the corresponding program from the storage unit 26 based on the user's operation.
In the following, unless otherwise specified, the holes 32 or the columnar portions 33 of the pallet 30 refer to the opening portion or the surface portion of the side surface 31 .

As shown in FIG. 3, the control unit 27 first acquires an image of the pallet 30 by the photographing unit 24 (step S1).
In this step, the control unit 27 operates the imaging unit 24 based on the user's operation to acquire an image of the palette 30 including depth information (an image including three-dimensional point cloud data). That is, in this step, a three-dimensional image including depth information and a corresponding RGB image (two-dimensional image) are obtained. The acquired image is stored in the storage unit 26 . As a result, an image of the palette 30 as shown in FIG. 4(a) is obtained.
Note that an image of the front of the forklift 1 may be acquired periodically at a predetermined photographing timing, and the pallet 30 may be detected from the image.

Next, the control unit 27 extracts a portion including the side surface 31 of the pallet 30 from the acquired image (RGB image (two-dimensional image) without depth information) as a target area (step S2).
This extraction method is not particularly limited, but a neural network is used in this embodiment. This is a neural network that has been trained to identify and output a rectangular area of the image that includes the side 31 of the pallet 30 from the captured image. As a result, as shown in FIG. 4B, a rectangular image portion G surrounding one side surface 31 of the palette 30 is extracted. If the image contains a plurality of side faces 31, for example, the side face 31 that is closest to the position directly facing the forklift 1 (that is, has the largest area) may be extracted, or the user (driver) can select it. may be
In this case, the image portion G including one side surface 31 is extracted as the target region. For example, it may be a rectangular range that includes the entire palette 30 .

Next, the control unit 27 extracts a local point group from the image portion G extracted as the target area in step S2 (step S3). That is, in this step, point cloud data included in the side surface 31 within the image portion G is extracted as depth information.

Next, the control unit 27 extracts a plane from the point cloud data extracted in step S3 (step S4). In this step, by fitting a plane to the point cloud data included in the side surface 31, a plane portion Ga corresponding to the side surface 31 is obtained as a characteristic portion of the pallet 30, as shown in FIG. The attitude of the pallet 30 is obtained by the orientation of the plane portion Ga. Here, the “characteristic portion” of the pallet 30 refers to a portion related to holding of the pallet 30 by the forks 12 .

Next, the control unit 27 extracts (detects) two holes 32 from the image portion G of the side surface 31 (step S5). Here, as a portion of the image portion G that does not contain the point cloud data, as shown in FIG. Then, the control unit 27 regards the obtained position of the hole region Gb as positive, and calculates and corrects the position of the entire pallet 30 as necessary.
Then, the controller 27 calculates the position of each hole region Gb (step S6). Here, the center position and the like of each hole region Gb are acquired.

Next, the control unit 27 compares the calculated position and orientation information of the pallet 30 (the side surface 31 and the hole 32) with the CAD data and corrects them (step S7).
In this step, the control unit 27 reads the CAD data of the pallet 30 from the storage unit 26 and generates, for example, a three-dimensional virtual image. Then, it is determined whether or not the two-dimensional silhouette image of the side face 31 matches the calculated information of the palette 30 (within a predetermined error), and if not matched, the calculated result is corrected to match. Further, the degree of matching between the calculation result and the CAD data is obtained, and if the degree of matching is less than a predetermined threshold, it is determined that the palette 30 of the image is different from the palette to be detected, and the display unit 23 may be displayed in

Next, the control section 27 outputs the calculation result (step S8).
In the present embodiment, whether or not the forks 12 can be inserted into the holes 32 is calculated (determined) from the calculated position/orientation information of the pallet 30 and displayed on the display unit 23 .
Specifically, for example, as shown in FIG. 6A, the current relative positions and attitudes of the fork 12 and the pallet 30 are displayed on the display unit 23 . Whether or not the fork 12 can be inserted is indicated by whether or not the predicted insertion line (extended line) L of the fork 12 properly penetrates the pallet 30 . In other words, “insertability” of the fork 12 into the hole 32 means whether the fork 12 is inserted into the hole 32 of the pallet 30 by moving the forklift 1 forward. This determination of "insertability" may include determining whether or not the orientation of the fork 12 and the orientation of the hole 32 (that is, the posture of the pallet 30) are within a predetermined range, or simply determining whether the tip of the fork 12 is It is also possible to judge only whether or not the object enters (the opening of) the hole 32 . The predicted insertion line L of the fork 12 is obtained by obtaining the height and tilt angle (position and posture) of the fork 12 from the encoder (or a predetermined sensor) of the cargo handling motor that drives the fork 12, and calculating the extension direction of the fork 12. do it. Alternatively, as shown in FIG. 6(b), this result display may be shown in a perspective view in which the holes 32 of the pallet 30 are visible.
In addition, the output of the calculation result is not limited to what is illustrated here. For example, if the calculation result indicates that the forks 12 are not properly inserted into the holes 32 of the pallet 30 (the predicted insertion line L of the forks 12 is out of the holes 32), the predicted insertion line L of the forks 12 is inserted into the holes 32. Adjustment amounts for the orientation of the forklift 1 and the height and angle of the forks 12 to fit within the portion 32 may be calculated and displayed on the display portion 23 to prompt adjustment. Furthermore, the driving section 21 may be controlled so that at least part of this adjustment is automatically performed. The automatic adjustment in this case may be performed only when the degree of comparison between the calculation result and the CAD data in step S7 is equal to or greater than a predetermined value. Also, the illustration of the forks 12 and the pallet 30 may be a plan view, a bird's-eye view, or a combination thereof, instead of the side view shown in FIG.
After that, the control unit 27 terminates the position/orientation calculation processing of the pallet 30 .

[Technical effect of the present embodiment]
As described above, according to this embodiment, the holes 32 are detected based on the two-dimensional image and the three-dimensional image of the pallet 30 .
Therefore, it is possible to improve the robustness of arithmetic processing compared to the conventional method based only on three-dimensional data.

Further, according to the present embodiment, the depth information (three-dimensional point cloud data) of the palette 30 is acquired from the image (three-dimensional image), and the target area including the hole 32 is extracted from the RGB image (two-dimensional image). . Then, the position and orientation of the pallet 30 are calculated based on the depth information of this target area.
As a result, the position and orientation of the pallet 30 can be calculated only by arithmetic processing of the three-dimensional point cloud data for the target area including the hole 32 in the acquired image. Therefore, the calculation cost can be reduced and the calculation rate can be improved as compared with the case where the calculation processing of the three-dimensional point cloud data is performed on the entire acquired image.

Further, according to the present embodiment, the planar portion Ga is extracted from the target area including the hole 32 as a characteristic portion related to holding the pallet 30, and the position and orientation of the pallet 30 are calculated based on the depth information.
Thereby, the calculation cost can be further reduced.

Further, according to the present embodiment, the display unit 23 displays whether or not the fork 12 can be inserted into the hole 32, so that the driver (operator) can determine whether or not the fork 12 can be inserted simply by looking at the display unit 23. can be determined immediately.

Further, according to this embodiment, the position and orientation of the pallet 30 calculated from the image are corrected based on the CAD data (shape information) of the pallet 30 .
As a result, the position and orientation of the pallet 30 can be calculated with higher accuracy.

[Modification]
Next, a modification of the above embodiment will be described.
This modification differs from the above-described embodiment in that the columnar portions 33 on both sides of the hole portion 32 of the side surface 31 are extracted when calculating the position/orientation of the pallet 30 . In the following, this difference will be mainly described, and the same reference numerals will be given to the same components as in the above embodiment, and the description will be omitted.
FIG. 7 is a flowchart showing the flow of the position/orientation calculation processing of the pallet 30 in this modification, and FIG. 8 is a diagram for explaining this processing.

As shown in FIG. 7, first, the control unit 27 acquires an image of the pallet 30 by the photographing unit 24 (step T1) in the same manner as in step S1 of the above embodiment.
Then, the control unit 27 extracts the image portion G including the side surface 31 of the palette 30 from the acquired image (RGB image that does not include depth information) as a target area (step S2 in the above embodiment) (step T2).

Next, the control unit 27 extracts a portion corresponding to the columnar portion 33 defining the hole portion 32 as a feature portion (local feature) from the image portion G extracted as the target region in step T2 (step T3 ).
This extraction method is not particularly limited, but a neural network is used in this embodiment. This is a neural network that has been trained to specify and output the image portion within the range in which the columnar portion 33 is shown from the image. As a result, three columnar regions Gc corresponding to the three columnar portions 33 are extracted from the image portion G, as shown in FIG. 8(a).

Next, the control unit 27 extracts a local point group from each columnar region Gc (step T4). That is, in this step, as shown in FIG. 8B, point cloud data included in each columnar region Gc is extracted as depth information.

Next, the control unit 27 obtains the center of gravity of each columnar region Gc from the point cloud data extracted in step T4 (step T5). Thereby, as shown in FIG. 8(c), the center-of-gravity vector Gd of each columnar region Gc is obtained.
Then, the control unit 27 calculates the position and orientation of each columnar region Gc from the center of gravity of each columnar region Gc (step T6). This also provides the position and orientation of the entire pallet 30 .

Next, the control section 27 extracts two holes 32 (hole regions Gb) based on the information of each columnar portion 33 (columnar region Gc) calculated in step T6 (step T7). However, substantially, the positions and orientations of the two holes 32 have already been determined at the stage of step T6 described above. It should be noted that the height of the hole portion 32 may be assumed to be the same as that of the columnar portion 33, or the upper and lower thickness portions of the hole portion 32 may be extracted and added. Then, the control unit 27 corrects the position of the entire pallet 30 as necessary, regarding the obtained position of the hole 32 as positive.

Next, the control unit 27 compares the calculated position and orientation information of the pallet 30 (the side surfaces 31, the hole portions 32, and the columnar portions 33) with the CAD data and corrects them in the same manner as in step S7 of the above embodiment. (Step T8).
Then, the control unit 27 outputs the calculation result (step T9) in the same manner as in step S8 of the above embodiment.
After that, the control unit 27 terminates the position/orientation calculation processing of the pallet 30 .

As described above, according to this modified example, the same effects as those of the above-described embodiment can be obtained.
Further, according to this modified example, as the characteristic portion of the pallet 30, the columnar region Gc corresponding to the columnar portion 33 defining the hole 32 is extracted, and based on the three-dimensional point cloud data of the columnar region Gc, The position and orientation of the pallet 30 are calculated.
As a result, the calculation cost can be further reduced and the calculation rate can be improved as compared with the above-described embodiment in which the planar portion Ga corresponding to the side surface 31 is extracted as the characteristic portion.

[others]
Although the embodiments of the present invention have been described above, the present invention is not limited to the above embodiments (including modifications).
For example, in the above embodiment, the calculation results of the position and orientation are corrected based on the CAD data of the pallet 30. A similar correction may be performed based on The size information of the holes 32 may be obtained from the CAD data of the pallet 30, or if there is no CAD data, for example, as shown in FIG. The size information may be displayed on the display unit 23 to allow the operator to input the size information. In this correction, for example, the hole portion 32 may be adjusted on the side surface 31 so that the point group of the columnar portion 33 does not enter the hole portion 32 the least.
Further, the information used for this correction does not have to be the size information of the hole 32 as long as it is the size information of the palette 30 that can be compared with the calculation result from the image. .

Further, in the above-described embodiment, the case where the pick-up assist device according to the present invention is mounted on the forklift 1 has been described, but it is not limited to this configuration, and for example, at least a portion of the pick-up assist device that performs calculation is It does not have to be mounted on the forklift 1. Specifically, for example, an arithmetic device (detection means) is provided in an external facility (for example, a control room of a cargo handling workplace), a communication means is provided between this external facility and the forklift 1, and the captured image is transmitted from the forklift 1 to the external facility. device, and the result calculated by this arithmetic device may be sent to the forklift 1. FIG.

In addition, the conveying apparatus according to the present invention is not limited to a forklift as long as it holds and conveys a cargo handling platform, for example, an unmanned guided vehicle (such as an AGV: Automated Guided Vehicle) that unmannedly conveys cargo. good too.
Moreover, the cargo handling platform according to the present invention is not limited to a pallet, and includes a gridiron, etc., as long as cargo is placed thereon and held by a conveying device.
In addition, the details shown in the above embodiments can be changed as appropriate without departing from the scope of the invention.

1 Forklift (conveyor)
12 fork (holding part)
23 display unit (display means)
24 photographing unit 26 storage unit 27 control unit (detection means)
30 pallets (loading platform)
31 Side 32 Hole 33 Column G Image portion Ga Plane portion Gb Hole region Gc Column region Gd Center of gravity vector L Fork insertion predicted line

Claims (11)

2D imaging means for acquiring a two-dimensional image of a loading platform having holes;
3D imaging means for acquiring a three-dimensional image of the loading platform;
detection means for detecting the hole of the loading platform based on the two-dimensional image and the three-dimensional image;
comprising
Pick-up support device. The detection means acquires depth information of the loading platform from the three-dimensional image, and calculates the position and orientation of the loading platform based on the depth information.
The pick-up support device according to claim 1. A target region extracting means for extracting a target region including the hole from the two-dimensional image,
The detection means is
Extracting a characteristic portion related to holding of the loading platform from the target area,
calculating the position and orientation of the loading platform based on the depth information of the characteristic portion;
The pickup support device according to claim 2. The characteristic portion is a planar portion that includes the hole, or a peripheral portion of the planar portion that defines the hole,
The pick-up support device according to claim 3. The target area extracting means extracts the target area using a neural network.
The pick-up support device according to claim 3 or 4. Acquiring means for acquiring information on the position and orientation of a holding unit that holds the loading platform,
The detection means is
calculating whether or not the holding portion can be inserted into the hole based on the position and orientation of the loading platform and the position and orientation of the holding portion;
The pickup support device according to any one of claims 2 to 5. Display means for displaying whether or not the holding portion can be inserted into the hole,
The pick-up support device according to claim 6. An information acquisition means for acquiring size information or shape information of the loading platform,
The detection means is
correcting the calculated position and orientation of the loading platform based on the size information or shape information of the loading platform acquired by the information acquiring means;
The pickup support device according to any one of claims 2 to 7. a pick-up support device according to any one of claims 1 to 8;
and a holding unit that holds the loading platform,
Conveyor. the control means
a 2D imaging step of acquiring a two-dimensional image of a loading platform having holes;
a 3D imaging step of acquiring a 3D image of the loading platform;
a detection step of detecting the hole of the loading platform based on the two-dimensional image and the three-dimensional image;
run the
Pick-up assistance method. the computer,
2D imaging means for acquiring a two-dimensional image of a loading platform having holes;
3D imaging means for acquiring a 3D image of the loading platform;
detection means for detecting the hole of the loading platform based on the two-dimensional image and the three-dimensional image;
to function as
pick-up assistance program.

Images