JP2024033761A - System and method for controlling cargo handling vehicle and cargo handling vehicle - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a system and a method for controlling a cargo handling vehicle capable of distinguishing between a target object and other objects to be checked when the cargo handling vehicle performs cargo handling work, and the cargo handling vehicle.

SOLUTION: A forklift 1 includes a three-dimensional sensor 51 mounted on the forklift 1 to detect an object existing around the forklift 1, a camera 52 mounted on the forklift 1 to capture the object exiting around the forklift 1, a controller 60 capable of acquiring object detection data from the three-dimensional sensor 51, and a display device 26. The controller 60 includes an identification unit 66 for identifying the target object from the other objects existing in a search range based on the object detection data from the three-dimensional sensor 51 and a display control unit 69 for allowing the display device 26 to display an image captured by the camera 52. The display control unit 69 allows the display device 26 to display an information image indicating the identified target object based on an identification result of the identification unit 66.

SELECTED DRAWING: Figure 3

COPYRIGHT: (C)2024,JPO&INPIT

Description

TECHNICAL FIELD The present disclosure relates to a system, method, and material handling vehicle for controlling a material handling vehicle.

BACKGROUND ART In the technical field related to cargo handling vehicles, a vehicle dispatch system for efficiently performing work with a forklift is known (see, for example, Patent Document 1). In Patent Document 1, a forklift is equipped with a camera and a display device, and a screen captured by the camera is displayed on the display device.

Japanese Patent Application Publication No. 2009-15684

In a work area where a cargo handling vehicle performs cargo handling work, there may be a plurality of objects for cargo handling work. Therefore, when carrying out cargo handling work, it is desirable to be able to distinguish between target objects and other objects.

Aspects of the present disclosure provide a system, a method, and a cargo handling vehicle for controlling a cargo handling vehicle that can distinguish and check a target object from other objects when the cargo handling vehicle performs cargo handling work. purpose.

According to an aspect of the present disclosure, an object detection sensor that is attached to a cargo handling vehicle and detects an object that exists around the cargo handling vehicle; a camera that is attached to the cargo handling vehicle and captures an image of an object that exists around the cargo handling vehicle; A system for controlling a cargo handling vehicle is provided that includes a controller capable of acquiring object detection data from an object detection sensor and a display device. The controller includes an identification unit that identifies a target from objects existing within a search range based on object detection data from an object detection sensor, and a display control unit that causes a display device to display an image captured by the camera. . The display control section causes the display device to display an information image indicating the identified object based on the identification result of the identification section.

According to an aspect of the present disclosure, an object detection sensor that is attached to a cargo handling vehicle and detects an object that exists around the cargo handling vehicle; a camera that is attached to the cargo handling vehicle and captures an image of an object that exists around the cargo handling vehicle; A method for controlling a cargo handling vehicle including a controller capable of acquiring object detection data from an object detection sensor and a display device is provided. The first step is to identify a target from objects existing within the search range based on object detection data from the object detection sensor. The second step is to display the image captured by the camera on the display device. The third step is to display an information image indicating the identified object on the display device based on the identification result.

According to an aspect of the present disclosure, an object detection sensor that is attached to a cargo handling vehicle and detects an object that exists around the cargo handling vehicle; a camera that is attached to the cargo handling vehicle and captures an image of an object that exists around the cargo handling vehicle; A cargo handling vehicle is provided that includes a controller capable of acquiring object detection data from an object detection sensor and a display device. The controller includes an identification unit that identifies a target from objects existing within a search range based on object detection data from an object detection sensor, and a display control unit that causes a display device to display an image captured by the camera. . The display control section causes the display device to display an information image indicating the identified object based on the identification result of the identification section.

According to aspects of the present disclosure, there is provided a system, a method, and a cargo handling vehicle for controlling a cargo handling vehicle that can distinguish and confirm a target object and other objects when the cargo handling vehicle performs cargo handling work. Ru.

FIG. 1 is a perspective view of a forklift according to an embodiment, viewed diagonally from the upper left front side. FIG. 2-1 is a schematic diagram showing the mounting position and pointing direction of the three-dimensional sensor according to the embodiment. FIG. 2-2 is a schematic diagram showing the mounting position and orientation direction of the three-dimensional sensor when the forklift is loaded according to the embodiment. FIG. 2-3 is a schematic diagram showing the mounting position and pointing direction of the three-dimensional sensor when the forklift is unloaded according to the embodiment. FIG. 3 is a functional block diagram showing an example of the configuration of the forklift according to the embodiment. FIG. 4 is a block diagram illustrating an example of a computer system according to an embodiment. FIG. 5 is a schematic diagram showing a first search range for searching for a target according to the embodiment. FIG. 6 is a schematic diagram showing a second search range for searching for a target according to the embodiment. FIG. 7 is a schematic diagram illustrating an example of a method for identifying a transported object according to an embodiment. FIG. 8 is a schematic diagram illustrating the positions of storage spaces that exist around objects to be transported according to the embodiment. FIG. 9 is a schematic diagram showing an example of a display on a display device when a storage space is identified, according to the embodiment. FIG. 10 is a schematic diagram illustrating an example of a display on the display device when the identified cargo storage space is determined as a target for carrying out cargo handling work, according to the embodiment. FIG. 11 is a schematic diagram showing an example of a display on a display device when a transported object is identified, according to the embodiment. FIG. 12 is a schematic diagram showing an example of the display on the display device when the identified cargo is determined as a target for cargo handling work, according to the embodiment. FIG. 13 is a flowchart illustrating an example of processing of the forklift control method according to the embodiment. FIG. 14 is a functional block diagram showing an example of the configuration of a remotely operated forklift according to a modification.

Hereinafter, embodiments according to the present disclosure will be described with reference to the drawings, but the present disclosure is not limited thereto. The components of each embodiment described below can be combined as appropriate. Furthermore, some components may not be used.

In the embodiment, the positional relationship of each part will be described using terms such as left, right, front, rear, upper, and lower. These terms indicate a relative position or direction with respect to the origin of the vehicle body coordinate system defined for the cargo handling vehicle.

<Cargo handling vehicle>
The cargo handling vehicle 1 identifies a target from objects existing around the cargo handling vehicle 1 at the work site of the cargo handling vehicle 1. In this embodiment, the cargo handling vehicle 1 is appropriately referred to as a forklift 1. Note that the cargo handling vehicle is not particularly limited, and may be, for example, a wheel loader to which a fork attachment can be attached.

The forklift 1 performs cargo handling operations such as loading and unloading of transported objects. For example, the forklift 1 performs a loading operation of picking up a cargo placed on the ground. For example, the forklift 1 performs a loading operation of picking up a cargo loaded on top of another cargo. For example, the forklift 1 performs a loading operation of unloading an article picked up by the fork 13 into a loading space. For example, the forklift 1 performs a loading operation of unloading an object picked up by the fork 13 onto the loading platform of a cargo vehicle. For example, the forklift 1 performs a loading operation in which an object picked up by the fork 13 is placed on top of other objects placed on the ground.

The object is the object to be handled by the forklift 1. In this embodiment, the target is, for example, an object to be transported, a storage space where the object can be placed, or a loading platform of a freight vehicle.

In this embodiment, the transported object is a loading platform or a container on which cargo is loaded. The conveyed object is, for example, a pallet, a skid, or a container. The conveyed object is equipped with a fork insertion hole.

The forklift 1 is configured to identify an object (carried object, cargo storage space, or loading platform of a freight vehicle) and automatically orient the forklift 1 directly toward the object during loading and unloading operations. Steering control is performed (automatic control function).

In the automatic control function, at least either the steering operation of the forklift 1 or the operation of the working machine is automatically controlled. In this embodiment, the operator operates the accelerator of the forklift 1. In this embodiment, the forklift 1 is manually moved by the operator's operation until it reaches the automatic control start area, in other words, until the object is identified.

<Overall configuration of forklift>
FIG. 1 is a perspective view of a forklift 1 according to an embodiment, viewed diagonally from the upper left front side. The forklift 1 includes a vehicle body 10, a power source 21, a working machine 2 disposed in front of the vehicle body 10, a traveling device that supports the vehicle body 10, and a driver's cab 20. The power source 21 drives a hydraulic pump (not shown). The power source 21 is, for example, an engine.

A working machine 2 is installed in front of the vehicle body 10 for picking up objects to be transported. The work machine 2 includes a mast 14 , a support portion 131 , a fork 13 , a lift cylinder 15 , a tilt cylinder 16 , and a side shift cylinder 17 . The mast 14 is attached to the front of the vehicle body 10 so as to be rotatable around the left and right axis. The mast 14 can be tilted forward or backward with respect to the vehicle body 10. The support part 131 is supported by the mast 14. The support part 131 is movable in the vertical direction along the mast 14. The fork 13 is supported by the mast 14 via a support portion 131. The fork 13 moves up and down along the mast 14.

The lift cylinder 15 moves the fork 13 vertically relative to the vehicle body 10. The tilt cylinder 16 tilts the mast 14 in the longitudinal direction with respect to the vehicle body 10. The side shift cylinder 17 moves the fork 13 in the left-right direction with respect to the vehicle body 10.

Lift cylinder 15 is arranged between mast 14 and support part 131. The lift cylinder 15 moves the fork 13 in the vertical direction by moving the support part 131 in the vertical direction. The support portion 131 and the fork 13 move together in the vertical direction. The support portion 131 and the fork 13 move in the vertical direction along the mast 14. The tilt cylinder 16 is arranged between the vehicle body 10 and the mast 14. The tilt cylinder 16 tilts the mast 14 in the front-back direction, thereby tilting the fork 14 in the front-back direction.

The traveling device causes the forklift 1 to travel. The traveling device moves, brakes, and steers the forklift 1. Advancing means that the forklift 1 moves forward or backward. Braking means that the forklift 1 decelerates or stops. Steering refers to changing the traveling direction of the forklift 1. The traveling device includes a front wheel 11, a rear wheel 12, a traveling motor 22, a brake device (not shown), and a steering cylinder 18.

Each of the front wheels 11 and the rear wheels 12 supports the vehicle body 10. At least a portion of the front wheel 11 is arranged below the vehicle body 10. At least a portion of the rear wheel 12 is arranged below the vehicle body 10. The front wheel 11 is arranged further forward than the rear wheel 12. The front wheels 11 are arranged on the left and right sides of the vehicle body 10, respectively. The rear wheels 12 are arranged on the left and right sides of the vehicle body 10, respectively. Each of the front wheel 11 and the rear wheel 12 is rotatable around a rotation axis.

The travel motor 22 generates driving force for moving the forklift 1. The travel motor 22 rotates the front wheels 11 to move the forklift 1 forward or backward. The travel motor 22 is driven by hydraulic oil discharged from a hydraulic pump (not shown). The front wheel 11 is a drive wheel that rotates by the rotational force generated by the travel motor 22. The brake device brakes the forklift 1.

The steering cylinder 18 steers the forklift 1. The steering cylinder 18 changes the running direction of the forklift 1 by steering the rear wheels 12. The rear wheel 12 is a steered wheel that is steered by a steering cylinder 18. The steering cylinder 18 is operated by hydraulic oil supplied through the steering control valve 24.

The display device 26 is arranged in the driver's cab 20. The display device 26 includes a flat panel display such as a liquid crystal display (LCD) or an organic electroluminescence display (OELD). The display device 26 displays the image transmitted from the controller 60.

The operating unit 27 is operated by the operator of the forklift 1. The operation unit 27 is arranged around the driver's seat in the driver's cab 20. The operation performed on the operation unit 27 is acquired by the controller 60 as an operation signal (operator command). The operation unit 27 may include a plurality of buttons, levers, switches, and the like.

The operation unit 27 receives, for example, an operation to permit the automatic control function. The operation unit 27 is, for example, a button, lever, switch, etc. that can accept an operation to permit an automatic control function. Target identification processing is performed while the automatic control function is enabled.

When an object is identified by the identification section 66, the operation section 27 receives an automatic control start operation for determining the identified object as a target for cargo handling work. When the automatic control start operation is accepted, the target identified at that time is determined as the target of the automatic control function. When the automatic control is started, at least one of the steering operation of the forklift 1 and the operation of the working machine is automatically controlled for cargo handling work on the determined target.

A steering wheel 31 is arranged in front of the driver's cab 20. The operator manually operates the steering wheel 31 to steer the forklift 1.

The forklift 1 includes a work machine lever 32, a forward/reverse switching lever 33, an accelerator pedal 34, a vehicle speed sensor 35, a steering angle sensor 36, and a work machine load sensor 37. The work machine lever 32 is an operating member for operating the fork 13 of the forklift 1. The forward/backward switching lever 33 is an operating member for switching the traveling direction of the forklift 1 to either the forward direction or the backward direction. The accelerator pedal 34 is an operating member for changing the traveling speed of the forklift 1.

Vehicle speed sensor 35 detects the vehicle speed of forklift 1. Vehicle speed sensor 35 transmits detected vehicle speed data to controller 60.

The steering angle sensor 36 detects the steering angle of the forklift 1. Steering angle sensor 36 transmits detected steering angle data to controller 60.

The work machine load sensor 37 detects the load on the work machine 2. The work machine load sensor 37 is, for example, a load measuring device such as a strain gauge or a load cell disposed on at least a portion of the work machine 2. Load data detected by the work machine load sensor 37 is output to the controller 60. Note that the load applied to the working machine 2 may be detected using, for example, a hydraulic sensor that detects the pressure of pressure oil that drives the lift cylinder 15. In this case, the load applied to the working machine 2 changes depending on whether the transported object is supported by the fork 13 or not. The work machine load sensor 37 detects changes in the load applied to the work machine 2.

<3D sensor and camera>
The forklift 1 includes a three-dimensional sensor 51, a camera 52, and a controller 60. The three-dimensional sensor 51, camera 52, and controller 60 are connected to be able to communicate data wirelessly or by wire.

The three-dimensional sensor 51 and camera 52 are attached to the forklift 1 as a set. In this embodiment, four sets of three-dimensional sensors 51 and cameras 52 are arranged on the forklift 1. Note that the number of three-dimensional sensors 51 and cameras 52 is not particularly limited; for example, two three-dimensional sensors 51 may be disposed on the forklift 1, or four or more cameras 52 may be disposed on the forklift 1. It's okay.

The three-dimensional sensor 51 acquires three-dimensional data of an object. The three-dimensional data of an object includes a point group consisting of a plurality of detection points defined on the surface of the object. The point group indicates the relative distance and position between the three-dimensional sensor 51 and each of the plurality of detection points defined on the surface of the object. As the three-dimensional sensor 51, a laser sensor (LiDAR: Light Detection and Ranging) that detects an object by emitting laser light is exemplified. Note that the three-dimensional sensor 51 may be an infrared sensor that detects objects by emitting infrared light or a radar sensor (RADAR: Radio Detection and Ranging) that detects objects by emitting radio waves.

The three-dimensional sensor 51 is an object detection sensor that detects objects existing around the forklift 1. The three-dimensional sensor 51 can detect objects around the forklift 1. The three-dimensional sensor 51 transmits detected object detection data to the controller 60. The three-dimensional sensor 51 can detect a range of, for example, several tens of meters. A plurality of three-dimensional sensors 51 are arranged on the forklift 1. At least one of the three-dimensional sensors 51 is arranged at a height close to the fork insertion hole of the transported object in order to identify the transported object. At least one of the three-dimensional sensors 51 is arranged, for example, at a height close to a fork insertion hole of a transported item arranged on the ground or on the loading platform of a freight vehicle. In this embodiment, the three-dimensional sensor 51 includes a three-dimensional sensor 51 ₁ , a three-dimensional sensor 51 ₂ , a three-dimensional sensor 51 ₃ , and a three-dimensional sensor 51 ₄ . The three-dimensional sensor 51 ₁ , the three-dimensional sensor 51 ₂ , the three-dimensional sensor 51 ₃ , and the three-dimensional sensor 51 ₄ are simply referred to as the three-dimensional sensor 51 when there is no particular need to distinguish them. The three-dimensional sensor 51 transmits detected object detection data to the controller 60.

The three-dimensional sensor ₅₁₁ detects the front right side of the forklift 1. The three-dimensional sensor ₅₁₁ is arranged above the fender 19R, which is located above the front wheel 11. The three-dimensional sensor ₅₁₁ is arranged on the right side of the lower front of the driver's cab 20. In this embodiment, the three-dimensional sensor 51 ₁ is arranged below the camera 52 ₁ .

The three-dimensional sensor ₅₁₂ detects the front left side of the forklift 1. The three-dimensional sensor ₅₁₂ is arranged above the fender 19L located above the front wheel 11. The three-dimensional sensor ₅₁₂ is arranged on the left side of the lower front of the driver's cab 20. In this embodiment, the three-dimensional sensor 51 ₂ is placed below the camera 52 ₂ .

The three-dimensional sensor ₅₁₃ detects the front of the forklift 1 when the forklift 1 is loaded. The three-dimensional sensor ₅₁₃ is arranged at a position where the front detection range is not blocked by the load on the fork 13 when the forklift 1 is loaded. The three-dimensional sensor ₅₁₃ is arranged on the fender 19L located above the front wheel 11. In this embodiment, the three-dimensional sensor 51 ₃ is placed below the camera 52 ₃ .

The three-dimensional sensor ₅₁₄ detects the lower part of the front center of the forklift 1 when the forklift 1 is empty. The three-dimensional sensor ₅₁₄ is arranged at a position where the front detection range is not blocked by the fork 13 when the forklift 1 is empty. The three-dimensional sensor ₅₁₄ is capable of detecting the fork insertion hole of a transported object placed on the ground or the like, or the fork insertion hole of a transported object loaded on the upper stage of another transported object, when the forklift 1 is empty. It is placed in a certain position. The three-dimensional sensor ₅₁₄ is arranged below the front central part of the vehicle body 10. The three-dimensional sensor ₅₁₄ is disposed below the center of a support portion 131 that is disposed at the rear of the fork 13 and supports the fork 13 with respect to the vehicle body 10. The three-dimensional sensor ₅₁₄ moves up and down together with the fork 13. In this embodiment, the three-dimensional sensor 51 ₄ is placed below the camera 52 ₄ .

FIG. 2-1 is a schematic diagram showing the mounting position and pointing direction of the three-dimensional sensor according to the embodiment. In the example shown in FIG. 2-1, a transported object 201R, a transported object 202R, and a transported object 203R are arranged on the right side of the forklift 1 in the order of distance from the forklift 1. On the left side of the forklift 1, a transported object 201L, a transported object 202L, and a transported object 203L are arranged in order of distance from the forklift 1. Reference numeral 101 schematically indicates the pointing direction and detection range of the three-dimensional sensor ₅₁₁ . Reference numeral 102 schematically indicates the pointing direction and detection range of the three-dimensional sensor ₅₁₂ . Reference numeral 103 schematically indicates the pointing direction and detection range of the three-dimensional sensor ₅₁₃ . Reference numeral 104 schematically indicates the pointing direction and detection range of the three-dimensional sensor ₅₁₄ . The forklift 1 can detect all directions in front of the forklift 1 using the three-dimensional sensor 51 , the three-dimensional sensor 51 ₁ , the three-dimensional sensor 51 ₂ , the three-dimensional sensor 51 ₃ , and the three-dimensional sensor 51 ₄ .

FIG. 2-2 is a schematic diagram showing the mounting position and orientation direction of the three-dimensional sensor when the forklift is loaded according to the embodiment. FIG. 2-2 shows the mounting position and pointing direction of the three-dimensional sensor ₅₁₃ . As shown in FIG. 2-2, when the forklift 1 is loaded, the detection range of the three-dimensional sensor ₅₁₃ is not obstructed by the object 200 to be transported. When the forklift 1 is loaded, the front of the forklift 1 can be detected by the three-dimensional sensor ₅₁₃ .

FIG. 2-3 is a schematic diagram showing the mounting position and pointing direction of the three-dimensional sensor when the forklift is unloaded according to the embodiment. FIG. 2-3 shows the mounting position and pointing direction of the three-dimensional sensor ₅₁₄ . As shown in the diagram on the left side of FIG. 2-3, when the forklift 1 is empty and the fork 13 is lowered, the lower part of the lower transported object 200 of the two-tiered transported object 200 is exposed to the three-dimensional sensor. Included in the detection range of ₅₁₄ . As shown in the right-hand diagram of FIG. Included in the detection range of ₄ . When the forklift 1 is empty, the front of the forklift 1 can be detected by the three-dimensional sensor ₅₁₄ .

The camera 52 images objects around the forklift 1. In this embodiment, the camera 52 is a monocular camera. Camera 52 has an optical system and an image sensor. The image sensor includes a CCD (Couple Charged Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor. The images captured by the camera 52 include a plurality of images per unit time depending on the frame rate.

The camera 52 is capable of capturing images around the forklift 1. The camera 52 is capable of capturing an image in a range of, for example, several tens of meters. A plurality of cameras 52 are arranged on the forklift 1. At least one of the cameras 52 is placed at a height close to the fork insertion hole of the conveyance to identify the conveyance. At least one of the cameras 52 is placed, for example, at a height close to a fork insertion hole of a conveyance placed on the ground or on the bed of a cargo vehicle. In this embodiment, the cameras 52 include a camera 52 ₁ , a camera 52 ₂ , a camera 52 ₃ , and a camera 52 ₄ . Camera 52 ₁ , camera 52 ₂ , camera 52 ₃ , and camera 52 ₄ are simply referred to as cameras 52 unless they need to be distinguished. The camera 52 transmits captured image data to the controller 60 as object detection data.

The camera ₅₂₁ images the front right side of the forklift 1. The camera ₅₂₁ is arranged above the fender 19R. The camera ₅₂₁ is disposed on the lower front right side of the driver's cab 20. In this embodiment, the camera 52 ₁ is placed above the three-dimensional sensor 51 ₁ .

The camera ₅₂₂ images the front left side of the forklift 1. The camera ₅₂₂ is placed above the fender 19L. The camera ₅₂₂ is placed on the left side of the lower front of the driver's cab 20. In this embodiment, the camera 52 ₂ is placed above the three-dimensional sensor 51 ₂ .

The camera ₅₂₃ images the front of the forklift 1 when the forklift 1 is loaded. The camera ₅₂₃ is arranged at a position where the front imaging range is not obstructed by the load on the fork 13 when the forklift 1 is loaded. The camera ₅₂₃ is placed on the fender 19L. In this embodiment, the camera 52 ₃ is placed above the three-dimensional sensor 51 ₃ .

The camera ₅₂₄ images the lower part of the front center of the forklift 1 when the forklift 1 is empty. The camera ₅₂₄ is arranged at a position where the front imaging range is not blocked by the fork 13 when the forklift 1 is empty. The camera ₅₂₄ is located at a position where it can image the fork insertion hole of a transported object placed on the ground or the like or the fork insertion hole of a transported object loaded on top of another transported object when the forklift 1 is empty. It is located in The camera ₅₂₄ is disposed below the front central portion of the vehicle body 10. The camera ₅₂₄ is disposed below the central portion of a support portion 131 that is disposed at the rear of the fork 13 and fixes and supports the fork 13 with respect to the vehicle body 10. The camera ₅₂₄ moves up and down together with the fork 13. In this embodiment, the camera 52 ₄ is placed above the three-dimensional sensor 51 ₄ .

<Forklift control system>
FIG. 3 is a functional block diagram showing an example of the configuration of the forklift according to the embodiment. The control system 70 for the forklift 1 includes a three-dimensional sensor 51 and a controller 60. The controller 60 identifies a target from objects existing within a search range around the forklift 1. The controller 60 can acquire object detection data from the three-dimensional sensor 51 as an object detection sensor. The controller 60 includes a numerical calculation device (processor) such as a CPU (Central Processing Unit).

FIG. 4 is a block diagram illustrating an example of a computer system according to an embodiment. Controller 60 includes computer system 1000. The computer system 1000 has a processor 1001 such as a CPU, a main memory 1002 including a nonvolatile memory such as a ROM and a volatile memory such as a RAM, a storage 1003, and an interface 1004 including an input/output circuit. The functions of the controller 60 described above are stored in the storage 1003 as a program. The processor 1001 reads the program from the storage 1003, expands it into the main memory 1002, and executes the above-described processing according to the program. Note that the program may be distributed to the computer system 1000 via a network.

In this embodiment, the controller 60 includes a sensor data acquisition section 61 , an image acquisition section 62 , a storage section 63 , a determination section 64 , an identification section 66 , and a display control section 69 .

The sensor data acquisition unit 61 acquires object detection data from the three-dimensional sensor 51. The object detection data acquired by the sensor data acquisition unit 61 is, for example, a point group consisting of a plurality of detection points.

The image acquisition unit 62 acquires image data captured by the camera 52. The image data acquired by the image acquisition unit 62 is image data for each frame.

The storage unit 63 includes at least one of RAM, ROM, flash memory, and hard disk drive. The storage unit 63 stores data and the like used in processing by the controller 60.

The storage unit 63 stores object dictionary data for identifying objects and a search range based on the origin of the vehicle body coordinate system of the forklift 1. In this embodiment, the storage unit 63 stores a first search range and a second search range different from the first search range. When there is no particular need to distinguish between the first search range and the second search range, they are simply referred to as search ranges.

The search range will be explained. When the forklift 1 performs automatic steering control to directly face an object, the range in which the forklift 1 can face directly without turning back is limited depending on the maximum curvature of the forklift 1 and the like. Therefore, when all objects within the range that can be detected by the three-dimensional sensor 51 are identified, some objects may be included that cannot be directly faced without turning back, and it may be difficult to face directly even if automatic steering control is started. There is. Furthermore, when a plurality of objects are identified during execution of the automatic control function, the operator has to spend more time selecting the object he or she wants to face directly from among the plurality of objects. In this way, identifying all objects within the range that can be detected by the three-dimensional sensor 51 not only increases processing load but also reduces convenience. Therefore, the search range is limited according to the range in which the forklift 1 can directly face the target without turning back. This prevents an increase in the processing load during execution of the automatic control function, and prevents the operator from being presented with an object that cannot be directly faced or a plurality of objects.

FIG. 5 is a schematic diagram showing a first search range 110 for searching for a target according to the embodiment. The first search range 110 is a search range for searching for the transported object 200 when the forklift 1 is empty. The first search range 110 has a rectangular shape. The first search range 110 has a first side 110a parallel to the front-rear direction and a second side 110b parallel to the left-right direction. The length of the first side 110a of the first search range 110 is Ly. The length of the second side 110b of the first search range 110 is Lx.

The first search range 110 is set at a position where the forklift 1 can face the target without turning back. In this embodiment, the first search range 110 is set at a position separated by a distance Rx in the left-right direction (to the left in FIG. 5) with respect to the origin of the vehicle body coordinates of the forklift 1. The first search range 110 is set at a position where the midpoint of the first side 110a is spaced forward by a distance Ry with respect to the origin of the vehicle body coordinates of the forklift 1. The first search range 110 is a position separated by a distance Rx and a distance Ry from the origin of the vehicle body coordinates of the forklift 1, and the first side 110a on the side near the forklift 1 (the right side in FIG. 5) and the first side 110a are perpendicularly bisected. The position of the intersection with the line is set to match. The origin of the vehicle body coordinate system of the forklift 1 is, for example, the intersection of the rotation axis of the front wheels 11 and the axis passing through the center of the vehicle body 10 in the width direction. The forklift 1 can directly face an object located at least a distance Rx in the left-right direction from the origin of the vehicle body coordinate system and at least a distance Ry forward from the origin of the vehicle body coordinate system without turning back.

In this embodiment, the length of the first side 110a of the first search range 110 is equal to the width of the front of the object to be transported. The front surface of the transported object is the surface directly facing the forklift 1 when picking up the transported object. By setting the length Ly of the first side of the first search range 110 to be the same as the width of the transported object, only one transported object among the plurality of transported objects lined up in the front-rear direction enters the first search range. Therefore, only one conveyed object can be identified from a plurality of conveyed objects. In this embodiment, the length of the second side 110b of the first search range 110 is longer than the first side, and the first search range 110 has a rectangular shape.

FIG. 6 is a schematic diagram showing a second search range 111 for searching for a target according to the embodiment. The second search range 111 is a search range for searching for a storage space when the forklift 1 is loaded. The second search range 111 has a rectangular shape. The second search range 111 has a first side 111a parallel to the front-rear direction and a second side 111b parallel to the left-right direction. The length of the first side 111a of the second search range 111 is 3Ly+2Cy. The length of the second side 111b of the second search range 111 is Lx.

The second search range 111 is set at a position where the forklift 1 can face the target without turning back. In this embodiment, the second search range 111 is set at a position separated by a distance Rx in the left-right direction (leftward in FIG. 6) with respect to the origin of the vehicle body coordinates of the forklift 1. The second search range 111 is set at a position where the midpoint of the first side 111a is spaced forward by a distance Ry with respect to the origin of the vehicle body coordinates of the forklift 1. The second search range 111 is a position separated by a distance Rx and a distance Ry from the origin of the vehicle body coordinates of the forklift 1, and a perpendicular bisection of the first side 111a on the side near the forklift 1 (the right side in FIG. 6) and the first side 111a. The position of the intersection with the line is set to match.

In this embodiment, the length of the first side 111a of the second search range 111 is longer than the length of the first side 110a of the first search range 110. In the example shown in FIG. 6, the second search range 111 includes area C, area F, and area B. The lengths of region C, region F, and region B in the front-rear direction are Ly. The length of each of the regions C, F, and B in the front-rear direction is equal to the length of the first side 110a of the first search range 110. The distance between region C and region F and the distance between region F and region B are Cy. Cy is, for example, the distance required when objects to be transported are placed adjacent to each other. Region C is a range in which the forklift 1 in the position shown in FIG. 6 can face directly without turning back. Region B is located further forward than the range in which the forklift 1 in the position shown in FIG. 6 can face directly without turning back. Region B is located further forward than region C. Region F is located further back than the range in which the forklift 1 in the position shown in FIG. 6 can face directly without turning back. Region F is located further back than region C. In this embodiment, the length of the first side 111a of the second search range 111 is longer than the second side 111b, and the second search range 111 has a rectangular shape.

The identification unit 66 identifies a target from objects existing within the search range based on object detection data from the three-dimensional sensor 51 as an object detection sensor. In this embodiment, the identification unit 66 identifies the object determined by the determination unit 65 that exists within the search range based on object detection data from the three-dimensional sensor 51 as an object detection sensor. For example, when the determining unit 64 determines that there is no cargo on the fork 13, the identifying unit 66 identifies the object to be transported from the object detection data. For example, when the determination unit 64 determines that there is a load on the fork 13, the identification unit 66 identifies the storage space from the object detection data. For example, when the determining unit 64 determines that there is a load on the fork 13, the identifying unit 66 identifies the loading platform of the freight vehicle from the object detection data.

First, identification of the transported object by the identification unit 66 will be explained. The identification unit 66 identifies objects to be transported within the search range from the object detection data.

FIG. 7 is a schematic diagram illustrating an example of a method for identifying a transported object according to an embodiment. The conveyed object shown in FIG. 7 is, for example, a box-shaped pallet. (a1) shows a point group that is object detection data obtained by irradiating a transported object with a laser beam from the three-dimensional sensor 51. (a2) shows the plane estimated from the point group of (a1) by hatching. The plane is estimated based on the relative positions of the three-dimensional sensor 51 and each of the plurality of detection points defined on the surface of the object, which are indicated by the point group. The estimated plane is a plane that directly faces the forklift 1 when the forklift 1 picks up the object. (a3) is a diagram obtained by converting the viewpoint of (a2) into an upward/downward view (planar view). In (a3), the plane estimated in (a2) is shown by a broken line. In (a3), a point group appears to the left of the plane estimated in (a2). The point group surrounded by an ellipse indicates the point group that passed through the plane estimated in (a2). In other words, the point group surrounded by an ellipse is the point group that has passed through the fork insertion hole of the transported object. (a4) is a diagram showing the intersection points of the estimated plane and a straight line connecting the point group passing through the plane and the coordinate system reference of the three-dimensional sensor 51. (a5) is a diagram in which the point group is complemented at the position of the intersection found in (a4). (a6) is a diagram in which the estimated plane is converted into a binary image viewed from the front. The boxed area shows the point group interpolated in (a5). The identification unit 66 refers to the target dictionary data stored in the storage unit 63, determines whether or not the supplemented point group is a feature indicating a fork insertion hole, and identifies the object to be transported.

For example, in a facility where transported items are temporarily stored, the transported items are placed adjacent to each other. For this reason, it is difficult to identify each conveyance object as an independent object from the point cloud that is the object detection data of the three-dimensional sensor 51. However, by identifying the fork insertion holes provided in each conveyance, it is possible to identify one conveyance from a group of adjacent conveyances.

Next, identification of cargo storage spaces by the identification unit 66 will be explained. The identification unit 66 searches for a space of a predetermined size existing within a predetermined range from the reference based on the object detection data from the three-dimensional sensor 51 as an object detection sensor, using the identified object as a reference, and searches for a space of a predetermined size within a predetermined range from the reference. identify. In this embodiment, the identification unit 66 identifies a storage space in which the object picked up by the forklift 1 can be placed, based on the object detection data and the object existing within the search range. More specifically, first, the identification unit 66 identifies objects to be transported that are present within the search range. Then, the identification unit 66 uses the identified transported object as a reference to search for a space of a predetermined size that exists around the reference transported object. In other words, the search unit 67 searches around the identified transport object for a space large enough to accommodate another transport object.

FIG. 8 is a schematic diagram showing the positions of storage spaces that exist around the transported object 200 according to the embodiment. The area around the conveyed object refers to the position adjacent to the conveyed object. As shown in FIG. 8, the periphery of the transported object 200 is the front of the transported object 200, the rear of the transported object 200, the right side of the transported object 200, and the upper side of the transported object 200. The space having a predetermined size is, for example, a space having a size larger than the size of the object to be transported. The identification unit 66 identifies the cargo within the search range, and identifies the storage space 140 based on a space of a predetermined size existing around the identified cargo. The identification unit 66 uses the transported object 200 located in the second search range 111 as a reference transported object, and identifies other transported objects that are large enough to place other transported objects that exist around the reference transported object 200 from the second search range 111. Explore the space.

In the present embodiment, the identification unit 66 identifies objects using object detection data from the three-dimensional sensor 51, but the invention is not limited thereto. For example, in another embodiment, the identification unit 66 may extract a feature amount from an image captured by the camera 52 and identify a target from the captured image based on the extracted feature amount and target dictionary data. good. The target identification method may be, for example, pattern matching or identification processing based on machine learning.

The determining unit 64 determines the target for cargo handling work based on the operation signal from the operating unit 27. More specifically, when the automatic control start operation is received by the operation unit 27, the determination unit 64 determines the object identified by the identification unit 66 at that time as the target for carrying out cargo handling work.

The display control unit 69 outputs an image signal that causes the display device 26 to display an image. The display control unit 69 outputs an image signal that causes the display device 26 to display the captured image captured by the camera 52.

The display control unit 69 outputs an image signal that causes the display device 26 to display an information image indicating the identified object based on the identification result of the identification unit 66. In the present embodiment, the display control unit 69 outputs an image signal so that the information image indicating the object identified by the identification unit 66 is displayed superimposed on the captured image captured by the camera 52. More specifically, based on the identification result of the identification unit 66, the display control unit 69 displays information of the first aspect indicating the object identified by the identification unit 66 at the position of the object on the captured image captured by the camera 52. It outputs an image signal on which the image is superimposed and displayed on the display device 26. The information image is an image that allows the operator to easily recognize the object identified by the identification unit 66. In this embodiment, the information image is a frame line representing the object. The information image may be, for example, a shape line representing the external shape of the identified object or a marker display suggesting the identified object.

The display control unit 69 outputs, for example, an image signal that causes the display device 26 to display the captured image captured by the camera 52 with a frame line of the first aspect representing the identified transported object superimposed thereon. For example, the display control unit 69 outputs an image signal that causes the display device 26 to display the image captured by the camera 52 with a frame line of the first aspect representing the identified cargo storage space superimposed thereon.

The display control unit 69 changes the form of the information image when the object identified by the identification unit 66 is determined by the determination unit 64 as a target for cargo handling work. In the present embodiment, when the object identified by the identification unit 66 is determined by the determination unit 64 as a target for cargo handling work, the display control unit 69 displays information in a second aspect different from the information image in the first aspect. The images are superimposed and displayed on the display device 26. More specifically, when the automatic control start operation is performed, the display control unit 69 displays a second aspect of displaying an object determined as a target for cargo handling work at the position of the object on the image taken by the camera 52. The frame lines are superimposed and displayed on the display device 6.

The first aspect and the second aspect differ, for example, in at least one of the color, line type, and thickness of the frame line. The first aspect and the second aspect are different from each other so that they can be easily distinguished by an operator.

FIG. 9 is a schematic diagram showing an example of a display on a display device when a storage space is identified, according to the embodiment. The display unit 160 of the display device 26 displays an image in which a frame line 161A of the first aspect indicating the storage space identified by the identification unit 66 is superimposed on the image captured by the camera 52. The frame line 161A shown in FIG. 9 is a thin broken line. As shown in FIG. 9, by displaying a frame line 161A superimposed on the position of the identified cargo storage space on the captured image, the operator can easily recognize the identified cargo storage space.

FIG. 10 is a schematic diagram illustrating an example of a display on the display device when the identified cargo storage space is determined as a target for carrying out cargo handling work, according to the embodiment. The display unit 160 of the display device 26 displays an image obtained by superimposing a frame line 161B of the second aspect indicating the storage space determined by the determining unit 68 on the image captured by the camera 52. The frame line 161B shown in FIG. 10 is a thick solid line. As shown in FIG. 10, by displaying a frame line 161B superimposed on the position of the identified cargo storage space on the captured image, the operator can easily recognize the cargo storage space determined as the target for cargo handling work. be able to.

FIG. 11 is a schematic diagram showing an example of a display on a display device when a transported object is identified, according to the embodiment. The display unit 160 of the display device 26 displays an image obtained by superimposing a frame line 161C of the first aspect indicating the transported object identified by the identification unit 66 on the image captured by the camera 52. A frame line 161C shown in FIG. 11 is a thin broken line. As shown in FIG. 11, by displaying a frame line 161C superimposed on the position of the identified transport object on the captured image, the operator can easily recognize the identified transport object.

FIG. 12 is a schematic diagram showing an example of the display on the display device when the identified cargo is determined as a target for cargo handling work, according to the embodiment. The display unit 160 of the display device 26 displays an image in which a frame line 161D of the second mode indicating the transported object determined by the determining unit 68 is superimposed on the image captured by the camera 52. The frame line 161D shown in FIG. 12 is a thick solid line. As shown in FIG. 12, by displaying a frame line 161D superimposed on the position of the identified transported object on the captured image, the operator can easily recognize the transported object determined as the target for cargo handling work. can.

In this embodiment, the display control unit 69 controls output to the display device 26. The display control unit 69 may control the display control unit 69 to distribute images to the display device 26 via a network (not shown).

<Processing by controller>
When the target identification process is started while the forklift 1 is being started, the process shown in FIG. 13 is executed.

<Basic processing>
FIG. 13 is a flowchart illustrating an example of processing of the forklift control method according to the embodiment. The controller 60 searches for a target (step ST11). More specifically, the controller 60 uses the identification unit 66 to search for a target among objects existing within the search range based on object detection data from the three-dimensional sensor 51 as an object detection sensor. The object to be searched from within the search range is a cargo to be transported, a storage space, or a loading platform of a cargo vehicle. The controller 60 proceeds to step ST12.

The controller 60 determines whether the target has been identified (step ST12). More specifically, the controller 60 determines whether the identification unit 66 has identified an object to be transported, a cargo storage space, or a loading platform of a cargo vehicle existing within the search range from the object detection data. When the identification unit 66 identifies a target from objects existing within the search range based on the object detection data (Yes in step ST12), the controller 60 proceeds to step ST13. In this case, the controller 60 causes the display control unit 69 to display the object identified by the identification unit 66 in the first mode at the position of the object on the image captured by the camera 52 based on the identification result of the identification unit 66. A frame line 161A, which is an information image, is superimposed and displayed on the display unit 160 of the display device 26. If the identification unit 66 does not identify a target from objects existing within the search range based on the object detection data (No in step ST12), the controller 60 executes the process in step ST11 again.

The controller 60 determines whether the operator has selected the identified object (step ST13). More specifically, the controller 60 determines whether an automatic control start operation from the operation unit 27 has been received. When the controller 60 receives the automatic control start operation from the operation unit 27 (Yes in step ST13), the controller 60 proceeds to step ST14. The processing from step ST14 to step ST22 is the processing of the automatic control function. When the controller 60 does not receive the automatic control start operation from the operation unit 27 (No in step ST13), the controller 60 executes the process of step ST11 again.

The controller 60 specifies the selected object as a target for cargo handling work (step ST14). More specifically, the controller 60 determines, by the determining unit 64, the object identified when the automatic control start operation is received as the target for performing cargo handling work. Then, the controller 60 causes the display control unit 69 to draw a frame line 161B, which is an information image of the second aspect, indicating the object determined as the target for cargo handling work, at the position of the object on the image captured by the camera 52. The images are displayed in a superimposed manner on the display unit 160 of the display device 26. The controller 60 proceeds to step ST15.

The controller 60 detects the distance from the origin of the vehicle body coordinate system to the target (step ST15). The controller 60 proceeds to step ST16.

The controller 60 generates a route to the target (step ST16). The controller 60 proceeds to step ST17.

The controller 60 performs control to follow the route (step ST17). More specifically, the controller 60 controls the forklift 1 to follow the route. Controller 60 controls the traveling device. In this embodiment, the controller 60 controls the steering cylinder 18 via the steering control valve 24. The controller 60 proceeds to step ST18.

The controller 60 determines whether the vehicle and the target are directly facing each other (step ST18). When the controller 60 determines that the vehicle and the target are directly facing each other (Yes in step ST18), the controller 60 proceeds to step ST19. When the controller 60 does not determine that the vehicle and the target are directly facing each other (No in step ST18), the controller 60 executes the process in step ST17 again.

The controller 60 detects the position of the target reference point in the vehicle body coordinate system (step ST19). More specifically, the controller 60 detects the position of a target reference point, which is a reference point of an object to be approached. The controller 60 proceeds to step ST20.

Controller 60 detects the position of work machine 2 (step ST20). The controller 60 proceeds to step ST21.

The controller 60 controls the work machine 2 so that the position of the target reference point and the position of the work machine 2 match (step ST21). More specifically, the controller 60 controls the work implement 2 so that the position of the work implement 2 coincides with the position of the target reference point. The controller 60 controls the lift cylinder 15 , tilt cylinder 16 , and side shift cylinder 17 via the work equipment control valve 23 . The controller 60 proceeds to step ST22.

The controller 60 determines whether the position of the target reference point and the position of the working machine 2 match (step ST22). If the controller 60 determines that the position of the target reference point and the position of the working machine 2 match (Yes in step ST22), the process ends. When the controller 60 determines that the position of the target reference point and the position of the working machine 2 do not match (Yes in step ST22), the controller 60 executes the process of step ST21 again.

<Effect>
As explained above, in this embodiment, a target is identified from objects existing within a search range based on object detection data from the three-dimensional sensor 51 that detects objects existing around the forklift 1, and the identification result is In this way, according to the present embodiment, an information image indicating the object identified in the image captured by the camera 52 is displayed on the display device 26. When making a decision, the operator can distinguish between the identified object and other objects.

In this embodiment, when a target object for carrying out cargo handling work is determined, a frame line in a second mode different from the first mode indicating the determined target is displayed superimposed on the position of the determined target. Displayed on the device 26. As described above, according to the present embodiment, when the cargo handling vehicle is automatically controlled to directly face the target for cargo handling work, the operator recognizes that the identified target is the target for cargo handling work. You can confirm that the decision has been made.

In this embodiment, the first aspect and the second aspect differ in at least one of the color, line type, and thickness of the frame line. According to this embodiment, when the automatic control start operation is performed, the operator can easily recognize the target determined as the target for cargo handling work by the display on the display device 26.

<Modified example>
FIG. 14 is a functional block diagram showing an example of the configuration of a remotely operated forklift according to a modified example. In the above-described embodiment, the display device 26 was described as being disposed in the driver's cab 20, but the display device 26 is not limited thereto. As shown in FIG. 14, for example, a remote control device and a remote display device may be provided in a monitoring room or the like located remotely away from the forklift 1.

1... Forklift (cargo handling vehicle), 10... Vehicle body, 11... Front wheel, 12... Rear wheel, 13... Fork, 14... Mast, 15... Lift cylinder, 16... Tilt cylinder, 17... Side shift cylinder, 18... Steering cylinder, 20... Driver's cab, 21... Power source, 22... Travel motor, 23... Work equipment control valve, 24... Steering control valve, 26... Display device, 27... Operating unit, 31... Steering wheel, 32... Work equipment lever, 33 ... Forward/forward switching lever, 34... Accelerator pedal, 35... Vehicle speed sensor, 36... Steering angle sensor, 37... Work equipment load sensor (load detection sensor), 51... Three-dimensional sensor (object detection sensor), 52... Camera, 60 ...Controller, 61...Sensor data acquisition section, 62...Image acquisition section, 63...Storage section, 64...Determination section, 66...Identification section, 69...Display control section.

Claims (9)

A system for controlling a cargo handling vehicle,
an object detection sensor that is attached to the cargo handling vehicle and detects objects existing around the cargo handling vehicle;
a camera that is attached to the cargo handling vehicle and captures images of objects existing around the cargo handling vehicle;
a controller capable of acquiring object detection data from the object detection sensor;
a display device;
Equipped with
The controller includes:
an identification unit that identifies a target from objects existing within a search range based on the object detection data from the object detection sensor;
a display control unit that causes the display device to display an image captured by the camera;
The display control unit causes the display device to display an information image indicating the identified object based on the identification result of the identification unit.
system. comprising an operation unit operated by an operator of the cargo handling vehicle,
The controller includes a determining unit that determines the object identified by the identifying unit as a target for cargo handling work based on an operation signal from the operating unit,
The display control unit changes an aspect of the information image when the target is determined by the determination unit as a target for cargo handling work.
The system of claim 1. The display control section includes:
superimposing the information image on the position of the object on the image taken by the camera and displaying it on the display device;
A system according to claim 1 or 2. The display control section includes:
When a target is identified by the identification unit, displaying an information image of a first aspect indicating the identified target on the display device;
When the target is determined by the determining unit as a target for carrying out cargo handling work, displaying on the display device an information image in a second aspect different from the first aspect, which shows the determined target;
The system according to claim 2. The information image of the first aspect and the information image of the second aspect are frame lines representing the object,
The system according to claim 4. The first aspect and the second aspect are different in at least one of the color, line type, and thickness of the frame line.
The system according to claim 5. an object detection sensor that is attached to a cargo handling vehicle and detects objects existing around the cargo handling vehicle;
a camera that is attached to the cargo handling vehicle and captures images of objects existing around the cargo handling vehicle;
a controller capable of acquiring object detection data from the object detection sensor;
a display device;
A method for controlling a cargo handling vehicle comprising:
identifying a target from objects existing within a search range based on the object detection data from the object detection sensor;
displaying an image captured by the camera on the display device;
displaying an information image indicating the identified object on the display device based on the identification result;
Method. determining the identified object as a target for cargo handling work based on an operation signal from an operation unit operated by an operator of the cargo handling vehicle;
changing the aspect of the information image when the target is determined as a target for cargo handling work;
The method according to claim 7. an object detection sensor that is attached to a cargo handling vehicle and detects objects existing around the cargo handling vehicle;
a camera that is attached to the cargo handling vehicle and captures images of objects existing around the cargo handling vehicle;
a controller capable of acquiring object detection data from the object detection sensor;
a display device;
Equipped with
The controller includes:
an identification unit that identifies a target from objects existing within a search range based on the object detection data from the object detection sensor;
a display control unit that causes the display device to display an image captured by the camera;
The display control unit causes the display device to display an information image indicating the identified object based on the identification result of the identification unit.
cargo handling vehicle.

Images