JP2023163605A - Cargo handling system - Google Patents

Abstract

To detect a position of a palette.SOLUTION: A cargo handling vehicle is equipped with a cargo handling device and an external sensor. The external sensor detects a position of an object using coordinates of a three-dimensional coordinate system. The control device detects positions of palettes and the number of the palettes, from point cloud data that is an assembly of points representing the position of the object. The control device determines the palette on which a cargo is placed.SELECTED DRAWING: Figure 4

Description

The present disclosure relates to cargo handling systems.

2. Description of the Related Art Patent Document 1 discloses a cargo handling vehicle that automatically picks up cargo under the control of a control device. The cargo handling vehicle disclosed in Patent Document 1 moves to a cargo picking position while estimating its own position. After moving to the loading position, the cargo handling vehicle picks up the load.

Japanese Patent Application Publication No. 2021-160885

When a cargo handling vehicle picks up cargo, it is necessary to move the cargo handling vehicle close to the pallet. In order to move the cargo handling vehicle close to the pallet, it is necessary to detect the position of the pallet.

A cargo handling system that solves the above problems is a cargo handling system that includes an external sensor that detects the position of an object using coordinates in a three-dimensional coordinate system, a cargo handling vehicle equipped with a cargo handling device, and a control device, the control device being , the position of the pallet is detected from point cloud data, which is a set of points representing the position of the object, and the pallet to be picked up is determined.

The points in the point cloud data represent the positions of objects. Therefore, the control device can detect the position of the pallet from the point cloud data.
In the cargo handling system, the control device creates a side view of the pallet from the point cloud data, and extracts the pallet from the side view by image recognition, thereby detecting the position of the pallet and the number of pallets. You may.

In the cargo handling system, the control device extracts the driver's seat of the transport vehicle loaded with the pallet from the side view by image recognition, and determines that the pallet furthest from the driver's seat is the pallet to be loaded. You may.

According to the present invention, the position of a pallet can be detected.

FIG. 2 is a schematic diagram of an area where cargo handling vehicles are operated. FIG. 2 is a perspective view of a transport vehicle and a cargo handling vehicle. FIG. 1 is a schematic configuration diagram of a cargo handling vehicle. 3 is a flowchart showing pallet detection control. It is a schematic diagram of a point cloud map. FIG. 3 is a schematic diagram of a side view.

An embodiment of the cargo handling system will be described below.
As shown in FIG. 1, a parking position PS1 is set in area A1. The transport vehicle 10 stops at the stop position PS1. The area of the stop position PS1 is wider than the area of the conveyance vehicle 10 when the conveyance vehicle 10 stopped at the stop position PS1 is viewed from above. There are no structures such as pillars at the parking position PS1. The area A1 is, for example, the whole or a part of a place such as a factory, a port, an airport, a commercial facility, or a public facility. In area A1, cargo handling vehicle 20 is operated. The cargo handling vehicle 20 performs loading and unloading. Loading is an operation in which the load C1 placed on the pallet PA1 is loaded onto the transport vehicle 10. Load picking is a task of picking up the pallet PA1 loaded on the transport vehicle 10 and the cargo C1 placed on the pallet PA1 from the transport vehicle 10. In the following description, the front, rear, left, right, top, and bottom of the transport vehicle 10 refer to the front, rear, left, right, top, and bottom when the transport vehicle 10 is used as a reference. The left, right, top, and bottom of the cargo handling vehicle 20 are the left, right, top, and bottom when the cargo handling vehicle 20 is used as a reference.

As shown in FIG. 2, the transport vehicle 10 is a wing truck. The transport vehicle 10 may be any type of truck, such as a flat body truck. The transport vehicle 10 includes a driver's seat 11, a front panel 12, a rear door 13, a loading platform 14, a wing side panel 16, and a gate 17.

The driver's seat 11 is a position where the driver of the transport vehicle 10 rides. The front panel 12 is provided behind the driver's seat 11 of the transport vehicle 10. The front panel 12 is provided adjacent to the driver's seat 11. The rear door 13 is provided behind the front panel 12 of the transport vehicle 10. The front panel 12 and the rear door 13 are spaced apart from each other in the longitudinal direction of the transport vehicle 10. The loading platform 14 extends in the longitudinal direction of the transport vehicle 10 between the front panel 12 and the rear door 13. The loading platform 14 includes a loading surface 15. The loading surface 15 is the upper surface of the loading platform 14. A load C1 placed on a pallet PA1 is loaded on the loading surface 15. The wing side panel 16 is provided between the front panel 12 and the rear door 13. The wing side panel 16 is provided so as to be rotatable in the vertical direction of the transport vehicle 10 around the center position of the transport vehicle 10 in the vehicle width direction. One wing side panel 16 is provided on each side of the transport vehicle 10 in the vehicle width direction. The gate 17 is provided so as to extend in the front-rear direction of the transport vehicle 10. The gate 17 is provided along the edge of the loading platform 14 and extending in the front-rear direction of the transport vehicle 10. One gate 17 is provided on each side of the transport vehicle 10 in the vehicle width direction.

<Cargo handling vehicle>
The cargo handling vehicle 20 includes a vehicle body 21 , drive wheels 22 , steering wheels 23 , and a cargo handling device 24 . The cargo handling device 24 is provided at the front of the vehicle body 21. The cargo handling device 24 includes a mast 25, a lift cylinder 28, a lift bracket 29, and a fork 30.

The mast 25 includes an outer mast 26 and an inner mast 27. The inner mast 27 is provided so as to be movable up and down relative to the outer mast 26. Fork 30 is fixed to lift bracket 29. The lift bracket 29 and the fork 30 move up and down together with the inner mast 27. The lift cylinder 28 moves the inner mast 27 up and down. Lift cylinder 28 is a hydraulic cylinder.

As shown in FIG. 3, the cargo handling vehicle 20 includes an external sensor 51, a control device 52, an auxiliary storage device 55, a vehicle control device 56, a traveling actuator 59, a cargo handling actuator 60, and a camera 61. . The cargo handling vehicle 20 is a cargo handling system.

The external sensor 51 detects the position of an object using coordinates in a three-dimensional coordinate system. The external sensor 51 is provided at the top of the cargo handling vehicle 20. For example, the external sensor 51 is provided on the head guard of the cargo handling vehicle 20.

Examples of the external sensor 51 include a millimeter wave radar, a stereo camera, a ToF (Time of Flight) camera, and a LIDAR (Laser Imaging Detection and Ranging). In this embodiment, LIDAR is used as the external sensor 51. The external sensor 51 irradiates the surrounding area with a laser and receives the reflected light from the point hit by the laser to derive the distance to the point. The point hit by the laser represents a portion of the object's surface. The position of a point can be represented by coordinates in a polar coordinate system. The coordinates of a point in a polar coordinate system are transformed to coordinates in a rectangular coordinate system. The conversion from the polar coordinate system to the orthogonal coordinate system may be performed by the external sensor 51 or by the control device 52. In this embodiment, it is assumed that the external sensor 51 is converting from a polar coordinate system to an orthogonal coordinate system. The external sensor 51 derives the coordinates of a point in the sensor coordinate system. The sensor coordinate system is a three-axis orthogonal coordinate system with the external world sensor 51 as the origin. The external sensor 51 outputs the coordinates of a plurality of points obtained by laser irradiation to the control device 52 as point group data.

The control device 52 includes a processor 53 and a storage section 54. The storage unit 54 includes a RAM (Random Access Memory) and a ROM (Read Only Memory). The storage unit 54 stores program codes or instructions configured to cause the processor 53 to execute processes. Storage 54, or computer readable media, includes any available media that can be accessed by a general purpose or special purpose computer. The control device 52 may be configured by a hardware circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). The control device 52, which is a processing circuit, may include one or more processors operating according to a computer program, one or more hardware circuits such as ASIC or FPGA, or a combination thereof.

The auxiliary storage device 55 stores information that can be read by the control device 52. Examples of the auxiliary storage device 55 include a hard disk drive and a solid state drive. The auxiliary storage device 55 stores an environmental map D1. The auxiliary storage device 55 stores the first image recognition model D2. The auxiliary storage device 55 stores a second image recognition model D3.

The environmental map D1 is information regarding the physical structure of the area A1, such as the shape of objects existing in the area A1 and the size of the area A1. In this embodiment, the environmental map D1 is data representing the structure of the area A1 using coordinates in a map coordinate system. The map coordinate system is a three-axis orthogonal coordinate system. The map coordinate system is a coordinate system whose origin is an arbitrary point in area A1. In the map coordinate system, the horizontal direction is defined by an X-axis and a Y-axis that are perpendicular to each other. It can be said that the XY plane defined by the X axis and the Y axis represents a horizontal plane. In the map coordinate system, the vertical direction is defined by the Z axis orthogonal to the X axis and the Y axis. The coordinates of the map coordinate system are appropriately referred to as map coordinates. The map coordinate system is a three-dimensional coordinate system that represents a three-dimensional position.

The control device 52 estimates the self-position of the cargo handling vehicle 20. The control device 52 outputs the self-position of the cargo handling vehicle 20 to the vehicle control device 56. The self-position is the position of the cargo handling vehicle 20 on the environmental map D1. The self-position is a coordinate indicating one point of the cargo handling vehicle 20 in the map coordinate system. Although one point on the cargo handling vehicle 20 is arbitrary, for example, the central position of the cargo handling vehicle 20 in the horizontal direction can be mentioned.

The self-position is estimated by comparing the detection results of the external sensor 51 and the environmental map D1. The control device 52 extracts landmarks having the same shape as the landmarks obtained from the point cloud data from the environmental map D1. The control device 52 recognizes the positions of landmarks from the environmental map D1. The positional relationship between the position of the landmark and the cargo handling vehicle 20 can be ascertained from the detection result of the external sensor 51. Therefore, the control device 52 can estimate its own position by recognizing the position of the landmark. A landmark is an object that has characteristics that can be identified by the external sensor 51. A landmark is a physical structure whose position is difficult to change. Landmarks include, for example, walls and pillars. Estimating the self-position may be performed by combining estimation of the self-position using the external sensor 51 and dead reckoning using the internal sensor. The self-position estimation may be performed by combining self-position estimation using the external sensor 51 and self-position estimation using a satellite signal transmitted from a GNSS (Global Navigation Satellite System) satellite.

The vehicle control device 56 includes, for example, the same hardware configuration as the control device 52. Vehicle control device 56 includes a processor 57 and a storage section 58.
The travel actuator 59 is an actuator that causes the cargo handling vehicle 20 to travel. Travel actuator 59 includes, for example, a motor that rotates drive wheels 22 and a steering mechanism. The vehicle control device 56 controls the traveling actuator 59 to cause the cargo handling vehicle 20 to travel while grasping its own position.

The cargo handling actuator 60 is an actuator that causes the cargo handling vehicle 20 to perform cargo handling. The cargo handling actuator 60 includes, for example, a motor that drives a pump that supplies hydraulic oil to hydraulic equipment, and a control valve that controls the supply of hydraulic oil. The hydraulic equipment includes a lift cylinder 28. The vehicle control device 56 raises and lowers the fork 30 by controlling the cargo handling actuator 60.

The cargo handling vehicle 20 automatically travels as the travel actuator 59 is controlled by the vehicle control device 56 . The cargo handling vehicle 20 automatically performs cargo handling by controlling the cargo handling actuator 60 by the vehicle control device 56. The cargo handling vehicle 20 is a self-driving forklift.

The camera 61 is arranged to image the front of the cargo handling vehicle 20. Camera 61 is a monocular camera. The camera 61 is provided, for example, between two forks 30 in the vehicle width direction of the cargo handling vehicle 20. The camera 61 is provided to move up and down together with the fork 30. For example, camera 61 is attached to lift bracket 29.

The pallet detection control performed by the control device 52 will be explained. The pallet detection control is performed in order to bring the cargo handling vehicle 20 closer to the pallet PA1 when the cargo handling vehicle 20 picks up cargo. The cargo handling vehicle 20 approaches the transport vehicle 10 based on, for example, a command from a higher-level control device. Then, when the distance between the cargo handling vehicle 20 and the transport vehicle 10 becomes less than a predetermined distance, pallet detection control is started. The predetermined separation distance is, for example, a distance that can be detected by the transport vehicle 10 by the external sensor 51.

<Pallet detection control>
As shown in FIGS. 4 and 5, in step S1, the control device 52 creates a point cloud map PM1. The point cloud map PM1 is obtained by superimposing point cloud data obtained from the detection results of the external sensor 51. The control device 52 acquires point cloud data from the external sensor 51 multiple times while the cargo handling vehicle 20 is moving. The control device 52 converts the coordinates of each point P1 of the point cloud data from the sensor coordinate system to map coordinates based on the self-position. From the self-position estimated by the control device 52, the origin of the sensor coordinate system in the map coordinate system can be recognized. From the self-position estimated by the control device 52, the deviation between the coordinate axes of the map coordinate system and the coordinate axes of the sensor coordinate system can be recognized. The control device 52 converts each point P1 of the point cloud data from the coordinates of the sensor coordinate system to the map coordinates based on the origin of the sensor coordinate system in the map coordinate system and the deviation between the coordinate axes of the map coordinate system and the coordinate axes of the sensor coordinate system. Convert to Each time the control device 52 acquires point cloud data, it creates a point cloud map PM1 by superimposing each point P1 converted into map coordinates. The point cloud map PM1 is a collection of point cloud data. The points P1 of the point cloud map PM1 are denser than the points P1 of the point cloud data.

FIG. 5 shows a point cloud map PM1 obtained by the process of step S1. Each point P1 included in the point cloud map PM1 represents the map coordinates of an object. For convenience of explanation, the explanation will be given by classifying each point P1 included in the point cloud map PM1 into a first point P11, a second point P12, a third point P13, and a fourth point P14. The first point P11 is a point P1 obtained by irradiating the tilt 17 with a laser. The second point P12 is a point P1 obtained by irradiating the driver's seat 11 with a laser. The third point P13 is a point P1 obtained by irradiating the pallet PA1 and the load C1 loaded on the loading surface 15 with a laser. The fourth point P14 is a point P1 that does not correspond to any of the first point P11, the second point P12, and the third point P13.

As shown in FIGS. 4 and 6, in step S2, the control device 52 creates a side view IM1 from the point cloud map PM1. Side view IM1 is a diagram in which point P1 is projected in the vehicle width direction of transport vehicle 10. As a result, the point cloud map PM1 can be made two-dimensional, so that the side view IM1 can be treated as image data. The side view IM1 can also be said to be a side view of the pallet PA1. An example of processing performed by the control device 52 when creating the side view IM1 will be described.

The control device 52 calculates the normal vector of each point P1. The normal vector is a vector directed in a direction perpendicular to the plane surrounded by the plurality of points P1. Examples of methods for deriving the normal vector include a method of finding a curved surface from each point P1 and deriving the normal vector of each point P1 from the curved surface, and a method of using the cross product of vectors. For example, when the control device 52 determines the normal vector of one point P1, it determines the cross product of vectors directed from this point P1 to each of two points P1 located within a predetermined range. This cross product is the normal vector.

The control device 52 extracts a point P1 whose normal vector is oriented in the horizontal direction. For example, the control device 52 determines whether the angle of the normal vector to the XY plane of the map coordinate system falls within a predetermined range. When the direction of the normal vector is horizontal, the normal vector is parallel to the XY plane of the map coordinate system. A predetermined range is set so that a point P1 where the direction of the normal vector can be regarded as horizontal can be extracted, taking into consideration the inclination of the transport vehicle 10 and measurement errors.

The control device 52 derives a plane equation from a point P1 whose normal vector is in the horizontal direction. The plane equation can be derived, for example, by using a robust estimation method such as RANSAC (Random Sample Consensus) or the least squares method. A plane represented by a plane equation is a plane that extends in the vertical direction. In the transport vehicle 10 of this embodiment, a flat surface can be obtained by the first point P11 caused by the tilt 17. Further, when picking up cargo, since the pallet PA1 and the cargo C1 are loaded on the transport vehicle 10, a flat surface can be obtained by the third point P13 of the pallet PA1 and the cargo C1.

The control device 52 creates a side view IM1 by projecting a point P1 in a direction perpendicular to a plane expressed by a plane equation. At this time, the control device 52 creates a side view IM1 by projecting points P1 in a predetermined range in the horizontal direction from a plane expressed by a plane equation. As shown in FIG. 1, pallets PA1 may be loaded on the transport vehicle 10 in a line in the width direction of the transport vehicle 10. The predetermined range PT1 is set so that among the pallets PA1 lined up in the vehicle width direction of the transport vehicle 10, a pallet PA1 having a long distance from the cargo handling vehicle 20 is not included. The predetermined range PT1 is set to include the surface facing the cargo handling vehicle 20 of the pallet PA1 that is shorter in distance from the cargo handling vehicle 20 among the pallets PA1 lined up in the vehicle width direction of the transport vehicle 10. When the cargo handling vehicle 20 picks up cargo, it picks up the pallet PA1 that is the shortest distance from the cargo handling vehicle 20 among the pallets PA1 lined up in the vehicle width direction of the transport vehicle 10. By setting the predetermined range PT1 as described above, it is possible to obtain the side view IM1 excluding the pallet PA1 that is not to be picked up by the cargo handling vehicle 20.

As shown in FIGS. 4 and 6, next, in step S3, the control device 52 detects the position of pallet PA1 and the number of pallets PA1. The position of pallet PA1 is the position of pallet PA1 that is to be picked up by cargo handling vehicle 20. The number of pallets PA1 is the number of pallets PA1 to be picked up by the cargo handling vehicle 20. The position of pallet PA1 is the coordinate of pallet PA1 in the map coordinate system. The position of pallet PA1 can be obtained by converting the coordinates of pallet PA1 in side view IM1 to coordinates in the map coordinate system. A detailed explanation will be given below.

Control device 52 determines the position of pallet PA1 in side view IM1 by extracting pallet PA1 from side view IM1. The position of pallet PA1 in side view IM1 can be represented by an image coordinate system. The image coordinate system is a coordinate system in which the horizontal direction of the side view IM1 is the X axis, and the vertical direction is the Y axis. The position of pallet PA1 is determined using image recognition. In this embodiment, a case will be described in which image recognition is performed using the first image recognition model D2, but image recognition may also be performed by pattern matching.

The first image recognition model D2 is a learned model generated by machine learning. The first image recognition model D2 uses an algorithm that can determine the class of an object on a region-by-region basis. "Palette" is set as a class. Examples of machine learning algorithms include SSD (Single Shot Multibox Detector), R-CNN (Regional Convolutional Neural Network), fast R-CNN, faster R-CNN, and YOLO (You Only Look Once). . The first image recognition model D2 is generated by supervised learning using teacher data or semi-supervised learning. As the teacher data, data including image data including an object corresponding to a class, the position of the object in the image data, and a label is used. The teacher data can be generated, for example, by surrounding an object in image data with a frame and attaching a label to the image data. In this embodiment, the image data palette PA1 may be surrounded by a frame and the image data labeled "palette" may be used as the teacher data. The image data used as the teacher data may be obtained from the point cloud map PM1 similarly to the side view IM1, or may be obtained by imaging with an imaging device.

The first image recognition model D2 specifies the area including the pallet PA1 from the input side view IM1. The area including palette PA1 is represented by bounding boxes B1 and B2. In the example shown in FIG. 6, two bounding boxes B1 and B2 are extracted from the side view IM1. The bounding boxes B1 and B2 surround the palette PA1 represented by the third point P13. The numbers of bounding boxes B1 and B2 represent the number of pallets PA1. The positions of bounding boxes B1 and B2 in side view IM1 represent the position of pallet PA1 in the image coordinate system. Since the side view IM1 is created by projecting the point P1 of the map coordinate system, the coordinates of the map coordinate system are associated with the points P1 forming the side view IM1. The control device 52 can detect the position of the pallet PA1 in the map coordinate system from the position of the pallet PA1 in the side view IM1. Since the side view IM1 is created from point cloud data, it can be said that the control device 52 detects the position of pallet PA1 and the number of pallets PA1 from the point cloud data.

Next, in step S4, the control device 52 determines the position of the driver's seat 11 in the side view IM1. The position of the driver's seat 11 in the side view IM1 is determined using image recognition. In this embodiment, a case will be described in which image recognition is performed using the second image recognition model D3, but image recognition may also be performed by pattern matching.

The second image recognition model D3 is a learned model generated by machine learning. The second image recognition model D3 is generated using the same method as the first image recognition model D2. In the second image recognition model D3, "driver's seat" is set as the class. Thereby, the second image recognition model D3 specifies the region including the driver's seat 11 from the input side view IM1. In the example shown in FIG. 6, a bounding box B3 is extracted from the side view IM1. In the example shown in FIG. 6, the driver's seat 11 represented by the second point P12 is surrounded by a bounding box B3. Bounding box B3 represents the position of driver's seat 11 in side view IM1.

Next, in step S5, the control device 52 determines the pallet PA1 to be picked up. If there are multiple pallets PA1, it can be said that the control device 52 determines the pallet PA1 to be picked up first. When the cargo handling vehicle 20 picks up cargo, it sequentially picks up the pallets PA1 loaded on the loading platform 14 of the carrier vehicle 10 from the rear to the front of the carrier vehicle 10. The control device 52 determines that the pallet PA1 farthest from the driver's seat 11 is the pallet PA1 to be picked up first. In this embodiment, the pallet PA1 represented by the bounding box B1 is the pallet PA1 to be picked up first. The pallet PA1 represented by the bounding box B2 is the second pallet PA1 to be picked up.

Next, in step S6, the control device 52 transmits the coordinates of the pallet PA1 in the map coordinate system, that is, the position of the pallet PA1 to the vehicle control device 56. The control device 52 may transmit only the position of the pallet PA1 to be picked up first to the vehicle control device 56. The control device 52 may transmit the detected positions of all pallets PA1 to the vehicle control device 56.

After completing the process of step S6, the control device 52 ends the pallet detection control.
Vehicle control device 56 controls travel actuator 59 according to the position of pallet PA1 transmitted from control device 52. The vehicle control device 56 generates a route to the pallet PA1 from which cargo is to be picked up first. At this time, the vehicle control device 56 generates a route to the approach start position a predetermined distance before the pallet PA1. The predetermined distance is set to a distance that allows the camera 61 to image the pallet PA1. The vehicle control device 56 controls the travel actuator 59 so that the cargo handling vehicle 20 moves following the route. When the cargo handling vehicle 20 reaches the approach start position, the vehicle control device 56 moves the cargo handling vehicle 20 closer to the pallet PA1 while confirming the position of the pallet PA1 using image data obtained by imaging with the camera 61. When the cargo handling vehicle 20 reaches the position where the pallet PA1 is to be picked up, the vehicle control device 56 controls the cargo handling actuator 60 to pick up the cargo. The vehicle control device 56 adjusts the position of the fork 30 in accordance with the pallet PA1 while confirming the position of the pallet PA1 based on image data obtained by imaging with the camera 61. Thereby, the cargo handling vehicle 20 can pick up the cargo.

[Effects of this embodiment]
(1) The control device 52 can detect the position of the pallet PA1 from point cloud data. Thereby, the control device 52 can determine the pallet PA1 to be picked up.

(2) The control device 52 can detect the position of the pallet PA1 from point cloud data. This allows vehicle control device 56 to generate a route toward pallet PA1. The camera 61 is used when the cargo handling vehicle 20 is moved from the approach start position to the position where the pallet PA1 is picked up. Since the detection range of the camera 61 is limited, it is required to accurately move the cargo handling vehicle 20 to the approach start position. By detecting the position of pallet PA1 from point cloud data and generating a route, cargo handling vehicle 20 can be accurately moved to the approach start position.

It is also conceivable to move the cargo handling vehicle 20 to a position where the pallet PA1 is to be picked up by setting a route in advance. However, in this case, it is necessary to set a route for each size of transport vehicle 10, size of pallet PA1, and type of pallet PA1, resulting in an enormous amount of man-hours. On the other hand, by detecting the position of pallet PA1 from point cloud data and generating a route, it is possible to generate a route according to the position of pallet PA1. Therefore, there is no need to set a route in advance.

(3) The control device 52 detects the position of the pallet PA1 and the number of pallets PA1 from the side view IM1 by image recognition. By converting the point cloud data into the side view IM1, the side view IM1 can be treated as image data. Thereby, the control device 52 can detect the position of the pallet PA1 in the side view IM1 and the number of pallets PA1 by image recognition. Then, the position of pallet PA1 in the map coordinate system can be detected from the position of pallet PA1 in side view IM1.

(4) The control device 52 extracts the driver's seat 11 from the side view IM1 by image recognition. When the cargo handling vehicle 20 picks up cargo, it sequentially picks up the cargo C1 loaded on the loading platform 14 of the carrier vehicle 10 from the rear to the front of the carrier vehicle 10. Since the driver's seat 11 of the transport vehicle 10 is located in the front of the transport vehicle 10, by extracting the driver's seat 11 of the transport vehicle 10, the pallet PA1 that is farthest from the driver's seat 11 is selected as the pallet PA1 to be loaded first. It can be determined that

(5) The position of pallet PA1 and the number of pallets PA1 can be detected using the external sensor 51 used to estimate the self-position. The number of parts can be reduced compared to the case where dedicated sensors are used to detect the position of pallet PA1 and the number of pallets PA1.

[Example of change]
The embodiment can be modified and implemented as follows. The embodiments and the following modification examples can be implemented in combination with each other within a technically consistent range.

The control device 52 may detect the position of the pallet PA1 and the number of pallets PA1 from the point cloud map PM1. In this case, for example, the position of pallet PA1 and the number of pallets PA1 are detected from point cloud map PM1 using a model that has learned the characteristics of three-dimensional data representing pallet PA1.

In step S3, the control device 52 only needs to be able to detect the position of the pallet PA1, and does not need to detect the number of pallets PA1.
- The control device 52 does not need to create the point cloud map PM1. In this case, the control device 52 converts the point cloud data acquired from the external sensor 51 into map coordinates, and performs the processing from step S2 onwards. That is, the control device 52 may detect the position of the pallet PA1 and the number of pallets PA1 from a single point cloud data without creating the point cloud map PM1 that is a collection of a plurality of point cloud data.

The vehicle control device 56 may generate a route to the position where the cargo handling vehicle 20 picks up the pallet PA1. In this case, image data obtained by imaging with the camera 61 is used to adjust the position of the fork 30.

If there is a space adjacent to the pallet PA1 in the horizontal direction, the control device 52 may determine that the pallet PA1 is the pallet PA1 to be picked up first. Regarding the pallet PA1 to be picked up first, there is no adjacent pallet PA1 in the rear direction of the transport vehicle 10. Therefore, on the pallet PA1 to be picked up first, there is a space that is equal to or larger than the threshold value and is adjacent to the pallet PA1 in the horizontal direction. By detecting this space, it is possible to determine which pallet PA1 is to be picked up first. When a cushioning material exists between adjacent pallets PA1, the threshold value is set in consideration of the thickness of the cushioning material. If there is no cushioning material between adjacent pallets PA1, the threshold value may be set based on the width of pallet PA1. For example, the threshold value may be half the width of the pallet PA1. The control device 52 may detect the space from the side view IM1 or may detect the space from the point cloud map PM1. Since the point P1 does not exist in the space, the control device 52 can detect the space depending on the presence or absence of the point P1.

As the second image recognition model D3, an algorithm that can determine the class of an object for each side view IM1 may be used. In this case, "rightward" and "leftward" may be set as the classes. An example of a machine learning algorithm is CNN (Convolution Neural Network). As the training data, it is sufficient to use image data in which the conveyance vehicle 10 facing right is labeled as "facing right" and image data in which the conveyance vehicle 10 facing left is labeled as "facing left". .

The control device 52 inputs the side view IM1 to the second image recognition model D3 to determine whether the conveyance vehicle 10 shown in the side view IM1 is facing right or left. When the conveyance vehicle 10 is facing right, the control device 52 determines that the pallet PA1 located furthest to the left in the image coordinate system is the pallet PA1 to be picked up. When the conveyance vehicle 10 is facing left, the control device 52 determines that the pallet PA1 located furthest to the right in the image coordinate system is the pallet PA1 to be picked up.

If the direction in which the transport vehicle 10 stops is fixed, the pallet PA1 to be picked up may be determined from the map coordinates of the pallet PA1. When the direction in which the conveyance vehicle 10 is stopped is constant, the farther back the pallet PA1 is loaded on the conveyance vehicle 10, the larger or smaller the coordinates in the horizontal direction of the map coordinate system. Whether the horizontal coordinates of the map coordinate system become larger or smaller as the pallet PA1 is loaded further back on the transport vehicle 10 depends on the positional relationship between the origin of the map coordinate system and the transport vehicle 10. The control device 52 can determine the pallet PA1 to be picked up based on the size of the map coordinates. For example, if the pallet PA1 loaded at the rear of the transport vehicle 10 has a larger X coordinate in the map coordinate system, the control device 52 selects the pallet PA1 that picks up the pallet PA1 with the largest X coordinate in the map coordinate system. can be determined.

The vehicle control device 56 may perform part of the pallet detection control. In this case, the vehicle control device 56 can also be said to be a control device.
A part of the pallet detection control may be performed by a host control device that issues commands to the cargo handling vehicle 20. In this case, the higher-level control device can also be said to be a control device. A host control device that issues commands to the cargo handling vehicle 20 may perform all of the pallet detection control processing. In this case, the higher-level control device is the control device. In this way, when the higher-level control device performs at least part of the processing of the pallet detection control, the control device 52 transmits data necessary for the processing to the higher-level control device by wireless communication. For example, when the higher-level control device performs all processing of pallet detection control, the control device 52 transmits point cloud data to the higher-level control device. The upper control device transmits the results obtained by performing the pallet detection control to the control device 52 by wireless communication. The upper control device is provided in area A1, for example. In this case, a cargo handling system is configured by the cargo handling vehicle 20 and the higher-level control device.

- The cargo handling vehicle 20 may pick up the pallet PA1 placed on a shelf or on the ground.
The transport vehicle 10 may be any vehicle that can perform transport, and may be, for example, an AGV (Automatic Guided Vehicle) or an AMR (Autonomous Mobile Robot).

The cargo handling vehicle 20 may be equipped with a laser distance meter instead of the camera 61 as a sensor for adjusting the position of the fork 30.
The cargo handling device 24 may be equipped with a robot arm, for example.

P1... point, PA1... pallet, 10... transport vehicle, 11... driver's seat, 20... cargo handling vehicle which is a cargo handling system, 24... cargo handling device, 51... external world sensor, 52... control device.

Claims (3)

A cargo handling vehicle equipped with an external sensor that detects the position of an object using coordinates in a three-dimensional coordinate system, and a cargo handling device;
A cargo handling system comprising a control device,
The control device includes:
Detecting the position of the pallet from point cloud data, which is a set of points representing the position of the object,
A cargo handling system that determines the pallet to be picked up. The control device includes:
creating a side view of the pallet from the point cloud data;
The cargo handling system according to claim 1, wherein the position of the pallet and the number of the pallets are detected by extracting the pallet from the side view by image recognition. The control device includes:
Extracting the driver's seat of the transport vehicle loaded with the pallet from the side view by image recognition,
The cargo handling system according to claim 2, wherein the pallet farthest from the driver's seat is determined to be the pallet to be picked up.

Images