US20240068201A1

US20240068201A1 - Work machine control system and work machine control method

Info

Publication number: US20240068201A1
Application number: US18/280,016
Authority: US
Inventors: Masataka Ozaki; Yohei Seki; Koji Kusaka; Yuki HOSODA
Original assignee: Komatsu Ltd
Current assignee: Komatsu Ltd
Priority date: 2021-03-19
Filing date: 2022-03-14
Publication date: 2024-02-29
Also published as: WO2022196641A1; CN116964279A; EP4283052A1; JP2022145209A

Abstract

A control system of a work machine includes: a three-dimensional measurement device 20 that measures a work target of a work machine 1; a detection device 25 that detects the work target; and a working equipment control unit 87 functioning as an intervention control unit that performs intervention control of the work machine 1 when both the three-dimensional measurement device 20 and the detection device 25 have detected the work target.

Description

FIELD

The present disclosure relates to a work machine control system and a work machine control method.

BACKGROUND

Patent Literature 1 discloses an example of a work machine capable of satisfactorily measuring a relative position with respect to a work target in order to implement automation of work by the work machine. In Patent Literature 1, a relative position between a wheel loader and a work target is measured based on measurement data obtained by a three-dimensional measurement device.

CITATION LIST

Patent Literature

Patent Literature 1: JP 2019-132068 A

SUMMARY

Technical Problem

In automation of work, it is desirable to detect the presence or absence of a work target by a work machine with high accuracy. However, a three-dimensional measurement device is susceptible to disturbance such as dust, rain, lighting, or direct sunlight.
An object of an aspect of the present disclosure is to detect the presence or absence of a work target with higher accuracy.

Solution to Problem

According to an aspect of the present invention, a control system of a work machine, the system comprises: a three-dimensional measurement device that measures a work target of the work machine; a detection device that detects the work target; and an intervention control unit that performs intervention control of the work machine when both the three-dimensional measurement device and the detection device detects the work target.
According to another aspect of the present invention, a method of controlling a work machine comprises performing intervention control of the work machine when both a three-dimensional measurement device that measures a work target of the work machine and a detection device that detects the work target detect the work target.

Advantageous Effects of Invention

According to the aspect of the present disclosure, the presence or absence of the work target can be detected with higher accuracy.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a side view illustrating an example of a work machine according to the present embodiment.

FIG. 2 is a schematic diagram illustrating motions of the work machine according to the present embodiment.

FIG. 3 is a schematic diagram illustrating a loading work mode of the work machine according to the present embodiment.

FIG. 4 is a functional block diagram illustrating a control system of a work machine according to the present embodiment.

FIG. 5 is a view illustrating operation of raising working equipment.

FIG. 6 is a diagram illustrating operation of loading an excavated object of the working equipment to a loading destination.

FIG. 7 is a view illustrating operation of lowering the working equipment.

FIG. 8 is a diagram illustrating an example of processing based on a determination result.

FIG. 9 is a flowchart illustrating a work machine control method according to the present embodiment.

FIG. 10 is a flowchart illustrating a method of detecting a work target by a detection device.

FIG. 11 is a block diagram illustrating an example of a computer system.

DESCRIPTION OF EMBODIMENTS

Hereinafter, embodiments according to the present disclosure will be described with reference to the drawings, but the present disclosure is not limited to the embodiments. The constituents described in the embodiments below can be appropriately combined with each other. In some cases, a portion of the constituents is not utilized. A control system of a work machine is a system that detects the presence or absence of a work target of the work machine with higher accuracy. The control system of the work machine is implemented by combining individual parts of the work machine.

Embodiment

[Wheel Loader]

FIG. 1 is a side view illustrating an example of a wheel loader 1 according to the present embodiment. A work machine 1 performs predetermined work toward a work target at a work site. In the present embodiment, the work machine 1 will be described as a wheel loader 1, which is a type of an articulated work machine. The predetermined work includes excavation work and loading work. The work target includes an excavation target and a loading target onto which the excavated object is loaded. The wheel loader 1 performs excavation work, which is work of excavating an excavation target and loading work, which is work of loading an excavated object excavated by the excavation work onto the loading target. The loading work is a concept including discharging work of discharging the excavated object to a discharge target. The excavation target includes, for example, at least one of natural hill, crag, coal, and a wall surface. The natural hill is a hill formed of earth and sand, and a crag is a hill formed of a rock or a stone. The loading target includes, for example, at least one of a haul vehicle, a predetermined area of a work site, a hopper, a belt conveyor, and a crusher.
As illustrated in FIG. 1 , the wheel loader 1 includes: a vehicle body 2, a cab 3 provided with a driver's seat; a traveling device 4 that causes the vehicle body 2 to travel; a transmission device 30; working equipment 10 supported by the vehicle body 2; an angle sensor 50 that detects an angle of the working equipment 10; a three-dimensional measurement device 20 that measures a work target in front of the vehicle body 2; a detection device 25 that detects a work target in front of the vehicle body 2; a buzzer 7 provided around the cab 3; a lamp 8 provided around the cab 3; and a control device 80.
The vehicle body 2 includes a vehicle body front portion 2F and a vehicle body rear portion 2R. The vehicle body front portion 2F and the vehicle body rear portion 2R are bendably connected to each other via a joint mechanism 9.
The cab 3 is supported by the vehicle body 2. At least a part of the wheel loader 1 is operated by a driver on the cab 3.
The traveling device 4 supports the vehicle body 2. The traveling device 4 can travel on a ground surface RS. The traveling device 4 has wheels 5. The wheels 5 are rotated by a driving force generated by an engine mounted on the vehicle body 2. The wheel 5 includes two front wheels 5F attached to the vehicle body front portion 2F and two rear wheels 5R attached to the vehicle body rear portion 2R. The wheels 5 are equipped with tires 6. The tires 6 include a front tire 6F attached to the front wheel 5F and a rear tire 6R attached to the rear wheel 5R. The front wheel 5F and the front tire 6F are rotatable about a rotation axis FX. The rear wheel 5R and the rear tire 6R are rotatable about the rotation axis RX. When the vehicle body 2 travels straight, the rotation axis FX and the rotation axis RX are parallel to each other.
In the following description, a direction parallel to the rotation axis FX of the front wheel 5F is appropriately referred to as a vehicle width direction. A direction orthogonal to the ground contact surface of the front tire 6F in contact with the ground surface RS is appropriately referred to as an up-down direction. A direction orthogonal to both the vehicle width direction and the up-down direction is appropriately referred to as a front-rear direction.
The traveling device 4 includes a driving device 4A, a braking device 4B, and a steering device 4C. The driving device 4A generates a driving force for accelerating the wheel loader 1. The driving device 4A includes an internal combustion engine such as a diesel engine, for example. The driving force generated by the driving device 4A is transmitted to the wheels 5 via the transmission device 30 to allow the wheels 5 to rotate. The braking device 4B generates a braking force for decelerating or stopping the wheel loader 1. The steering device 4C can adjust the traveling direction of the wheel loader 1. The traveling direction of the wheel loader 1 includes the direction of the vehicle body front portion 2F. The steering device 4C bends the vehicle body front portion 2F by a hydraulic cylinder, thereby adjusting the traveling direction of the wheel loader 1.
In the present embodiment, the traveling device 4 is operated by a driver on the cab 3. The cab 3 is equipped with a travel operation device 40 which is used to operate the traveling device 4. The driver operates the travel operation device 40 to activate the traveling device 4. The travel operation device 40 includes an accelerator pedal, a brake pedal, a steering lever, and a gear shift lever 41 for switching forward and backward movements. Operation on the accelerator pedal increases the traveling speed of the wheel loader 1. Operation on the brake pedal decreases the traveling speed of the wheel loader 1 or stops traveling of the wheel loader 1. Operation on the steering lever causes the wheel loader 1 to swing. Operation on the gear shift lever 41 switches forward or backward movement of the wheel loader 1.
The transmission device 30 transmits the driving force generated by the driving device 4A to the wheels 5.
The working equipment 10 is controlled by the control device 80. The working equipment 10 includes: a boom 11 pivotably connected to the vehicle body front portion 2F; and a bucket 12 pivotably connected to the boom 11.
The boom 11 is activated by power generated by a boom cylinder 13. Expansion or contraction of the boom cylinder 13 causes the boom 11 to perform a raising motion or a lowering motion. The boom cylinder 13 includes a boom control valve (not illustrated) that controls a flow rate and a direction of hydraulic oil supplied from a hydraulic pump (not illustrated).
The bucket 12 is a work member having a distal end 12B including a blade edge. The bucket 12 is disposed in front of the front wheel 5F. The bucket 12 is connected to a distal end of the boom 11. The bucket 12 is connected to the bucket cylinder 14 via a bell crank 15 and a link 16. The bucket 12 is activated by power generated by the bucket cylinder 14. The bucket cylinder 14 includes a bucket control valve (not illustrated) that controls the flow rate and the direction of the hydraulic oil supplied from the hydraulic pump. Expansion or contraction of the bucket cylinder 14 causes the bucket 12 to perform a dumping motion or a tilting motion. The dumping motion causes the excavated object in the bucket 12 to be discharged from the bucket 12. The tilting motion causes the bucket 12 to scoop up the excavated object.
The angle sensor 50 is mounted on the working equipment 10 and detects the posture of the working equipment 10. The angle sensor 50 detects an angle of the working equipment 10. The angle sensor 50 includes a boom angle sensor 51 that detects the angle of the boom 11 and a bucket angle sensor 52 that detects the angle of the bucket 12. The boom angle sensor 51 detects an angle of the boom 11 with respect to a reference axis of a vehicle body coordinate system defined in the vehicle body front portion 2F, for example. The bucket angle sensor 52 detects an angle of the bucket 12 with respect to the boom 11. The angle sensor 50 may be a potentiometer, a stroke sensor that detects a stroke of the hydraulic cylinder, an inertial measurement unit, or an inclinometer. Angle data indicating the angle of the working equipment 10 is output to a position data calculation unit 83 and a determination unit 91 described below.
The three-dimensional measurement device 20 is mounted on the wheel loader 1. The three-dimensional measurement device 20 measures a work target in front of the vehicle body front portion 2F. The work target includes a loading target onto which the excavated object excavated by the working equipment 10 is loaded. The three-dimensional measurement device 20 measures the relative position from the three-dimensional measurement device 20 to each of a plurality of measurement points on the surface of the work target, thereby measuring the three-dimensional shape of the work target. The three-dimensional measurement device 20 includes a stereo camera 22 which is a type of a photographic measurement device. The stereo camera 22 is disposed on either side, namely, the right side and the left side in the vehicle width direction of the vehicle body 2. In the following description, the stereo camera 22 on one side will be described.
The stereo camera 22 captures an image of the front. The stereo camera 22 captures an image of a work target and measures a work target. In the present embodiment, the stereo camera 22 measures a work target including at least a loading target such as a haul vehicle LS. The measurement data of the stereo camera 22 includes image data of a work target. The image data includes a plurality of pixels. The image data is an example of measurement data.
The stereo camera 22 includes a first camera 22A and a second camera 22B, as a pair of cameras. The first camera 22A and the second camera 22B are spaced apart from each other. First image data acquired by the first camera 22A and second image data acquired by the second camera 22B are output to the control device 80. The first image data and the second image data are two-dimensional image data. The detection device 25 is mounted on the wheel loader 1. The detection device 25 is disposed at a position different from the position of the three-dimensional measurement device 20. The detection device 25 detects a detection target in front of the vehicle body front portion 2F. The detection device 25 measures a three-dimensional shape of a detection target. The detection device 25 includes a non-contact sensor 26. The non-contact sensor 26 is disposed in the wheel loader 1. The non-contact sensor 26 perform non-contact detection of an object around the wheel loader 1. The non-contact sensor 26 scans the surroundings of the wheel loader 1 to detect an object. The non-contact sensor 26 includes a radar device that detects an object by scanning the surroundings of the wheel loader 1 with radio waves such as millimeter waves. The detection data of the non-contact sensor 26 includes presence/absence data indicating the presence or absence of an object and position data indicating the position of the object. Detection data obtained by the non-contact sensor 26 is output to the control device 80.
The buzzer 7 is disposed in the vicinity of the cab 3. The buzzer 7 is a buzzer device that outputs a warning sound. The buzzer 7 outputs a determination result of the determination unit 91. The buzzer 7 outputs a warning sound in a case where the determination unit 91 determines that either one of the devices has detected the haul vehicles LS.
The lamp 8 is disposed in the vicinity of the cab 3. The lamp 8 outputs a determination result of the determination unit 91. The lamp 8 is set such that, in a case where the determination unit 91 determines that either one of the devices has detected the haul vehicle LS, the lamp 8 is set to blink on/off. The lamp 8 is set such that, in a case where the determination unit 91 determines that both devices have detected the haul vehicle LS, the lamp 8 is turn to steady-on. The lamp 8 is set such that, in a case where the determination unit 91 determines that neither device has detected the haul vehicle LS, the lamp 8 is set to steady-off.
[Operation]
FIG. 2 is a schematic diagram illustrating operation of the wheel loader 1 according to the present embodiment. The wheel loader 1 works in a plurality of work modes. The work mode includes: an excavation work mode in which the bucket 12 of the working equipment 10 excavates an excavation target; and a loading work mode in which the excavated object scooped up by the bucket 12 in the excavation work mode is loaded onto a loading target. An example of the excavation target is a natural hill DS on the ground surface RS. An example of the loading target is a vessel BE of the haul vehicle LS capable of traveling on the ground surface RS. An example of the haul vehicle LS is a dump truck.
In the excavation work mode, the wheel loader 1 advances toward the natural hill DS in a state where the excavated object is not held by the bucket 12. The driver operates the travel operation device 40 to move the wheel loader 1 forward to approach the natural hill DS as indicated by an arrow M1 in FIG. 2 . The control device 80 controls the working equipment 10 to excavate the natural hill DS by the bucket 12. This causes the natural hill DS to be excavated by the bucket 12, and causes the excavated object to be scooped up by the bucket 12.
The wheel loader 1 moves backward so as to be separated away from the natural hill DS in a state where the excavated object is held by the bucket 12. The driver operates the travel operation device 40 to move the wheel loader 1 backward and separate the wheel loader 1 from the natural hill DS as indicated by an arrow M2 in FIG. 2 .
Next, a loading work mode is implemented. In the loading work mode, the wheel loader 1 advances toward the haul vehicle LS in a state where the excavated object is held by the bucket 12. The driver operates the travel operation device 40 to move the wheel loader 1 forward with swing operation to approach the haul vehicle LS as indicated by an arrow M3 in FIG. 2 . At this time, the three-dimensional measurement device 20 mounted on the wheel loader 1 measures the haul vehicle LS. The control device 80 controls the working equipment 10 so as to load the excavated object held by the bucket 12 onto the vessel BE of the haul vehicle LS based on the measurement data of the three-dimensional measurement device 20. That is, the control device 80 controls the working equipment 10 to cause the boom 11 to take a raising motion in a state where the wheel loader 1 moves forward so as to approach the haul vehicle LS. After the boom 11 takes a raising motion and the bucket 12 is disposed above the vessel BE, the control device 80 controls the working equipment 10 to allow the bucket 12 to take a dumping motion. The excavated object is discharged from the bucket 12 that has taken the dumping motion, and then loaded into the vessel BE.
After the excavated object is loaded onto the vessel BE, the wheel loader 1 moves backward so as to be separated from the haul vehicle LS in a state where the excavated object is not held by the bucket 12. The driver operates the travel operation device 40 to move the wheel loader 1 backward with swing operation to be separated away from the haul vehicle LS as indicated by an arrow M4 in FIG. 2 .
The driver and the control device 80 repeat the above operations until the vessel BE is fully loaded with the excavated object or the excavation of the natural hill DS is completed.
FIG. 3 is a schematic diagram illustrating a loading work mode of the wheel loader 1 according to the present embodiment. The driver operates the travel operation device 40 to move the wheel loader 1 forward with swing operation to approach the haul vehicle LS. As illustrated in FIG. 3(A), the three-dimensional measurement device 20 measures the three-dimensional shape of the haul vehicle LS and the relative position with respect to the haul vehicle LS. Based on the measurement data obtained by the three-dimensional measurement device 20, the control device 80 detects a distance db between the wheel loader 1 and the haul vehicle LS together with a height Hb of an upper end BEt of the vessel BE.
As illustrated in FIG. 3(B), in a state where the wheel loader 1 is moving forward to approach the haul vehicle LS, the control device 80 controls the boom 11 to take a raising motion while controlling the angle of the bucket 12 so that the bucket 12 is disposed above the upper end BEt of the vessel BE and the excavated object held by the bucket 12 does not spill from the bucket 12 based on the measurement data of the three-dimensional measurement device 20.
As illustrated in FIG. 3(C), after the boom 11 takes the raising motion to set the bucket 12 above the vessel BE, the control device 80 controls the working equipment 10 to cause the bucket 12 to take a dumping motion. With this control, the excavated object is discharged from the bucket 12 and then loaded into the vessel BE.
After the state of FIG. 3(C), the driver operates the travel operation device 40 to move the wheel loader 1 backward with swing operation to be separated away from the haul vehicle LS.
[Control Device]
FIG. 4 is a functional block diagram illustrating a control system 200 of the wheel loader 1 according to the present embodiment. The control device 80 includes a computer system. The control device 80 controls the wheel loader 1. The control device 80 is connected to the working equipment 10, the three-dimensional measurement device 20, the detection device 25, the angle sensor 50, the travel operation device 40, the buzzer 7, and the lamp 8. The control device 80 includes a measurement data acquisition unit 81, a detection data acquisition unit 84, a storage unit 82, a position data calculation unit 83, a target calculation unit 86, a working equipment control unit 87 as an intervention control unit, a determination unit 91, and an output control unit 92. The buzzer 7 is an example of an output unit. The lamp 8 is an example of an output unit. The working equipment control unit 87 is an example of an intervention control unit. The position data calculation unit 83 is an example of a position calculation unit.
The control system 200 is an example of an abnormality determination system. The control system 200 includes the working equipment 10, the three-dimensional measurement device 20, the angle sensor 50, the travel operation device 40, the buzzer 7, the lamp 8, and the control device 80.
The control device 80 calculates a parameter related to the loading target based on the measured three-dimensional shape of the loading target. The parameter related to the loading target includes at least one of the distance to the loading target, the position of the upper end of the loading target, or the height of the loading target. The control device 80 performs intervention control on the working equipment 10 based on the calculated parameters.
The measurement data acquisition unit 81 acquires measurement data of the three-dimensional measurement device 20. In the present embodiment, the measurement data acquisition unit 81 acquires the first image data from the first camera 22A of the stereo camera 22, and acquires the second image data from the second camera 22B. The image data of the work target acquired by the measurement data acquisition unit 81 is output to the target calculation unit 86.
The detection data acquisition unit 84 acquires detection data of the detection device 25. In the present embodiment, the detection data acquisition unit 84 acquires detection data from the non-contact sensor 26. The determination method of the presence or absence of the haul vehicle LS including the vessel BE from the detection data is implemented by using a known method, and an example of the method will be described below with reference to FIG. 10 . The detection data acquired by the detection data acquisition unit 84 is output to the determination unit 91.
In the present embodiment, the detection data acquisition unit 84 can detect the presence or absence of the haul vehicle LS based on whether the detection point exists within a predetermined region. The predetermined region is a region at a position where the haul vehicle LS is estimated to be present within a scanning range of the non-contact sensor 26. The predetermined region may be defined according to the relative positional relationship between the wheel loader 1 and the haul vehicle LS or the distance db between the wheel loader 1 and the haul vehicle LS, for example.
The storage unit 82 stores working equipment data. The working equipment data is data regarding the working equipment 10, which specifically includes: design data including computer aided design (CAD) data; or specification data, for example. The working equipment data includes outer shape data including dimensional data of the working equipment 10.
In the present embodiment, the working equipment data includes a boom length, a bucket length, and a bucket outer shape. The boom length refers to a distance between the boom rotation axis and the bucket rotation axis. The bucket length refers to a distance between the bucket rotation axis and the distal end 12B of the bucket 12. The boom rotation axis refers to a rotation axis of the boom 11 with respect to the vehicle body front portion 2F, and includes a connecting pin that connects the vehicle body front portion 2F and the boom 11 to each other. The bucket rotation axis refers to a rotation axis of the bucket 12 with respect to the boom 11, and includes a connecting pin that connects the boom 11 and the bucket 12 to each other. The bucket outer shape includes the shape and dimensions of the bucket 12. The dimensions of the bucket 12 include a bucket width indicating a distance between a left end and a right end of the bucket 12, a height of an opening of the bucket 12, a bucket bottom surface length, and the like.
The position data calculation unit 83 calculates position data indicating the posture of the working equipment 10 based on the detection result from the angle sensor 50. More specifically, the position data calculation unit 83 calculates the position data of the working equipment 10 based on the angle data of the working equipment 10 detected by the angle sensor 50 and the working equipment data of the working equipment 10 stored in the storage unit 82. The position data of the working equipment 10 includes position data of each portion of the bucket 12 in the vehicle body coordinate system, for example. The position data of the working equipment 10 calculated by the position data calculation unit 83 is output to the determination unit 91.
The target calculation unit 86 calculates the position of the loading target of the wheel loader 1 based on the measurement result of the three-dimensional measurement device 20. More specifically, the target calculation unit 86 calculates three-dimensional data of the work target measured by the three-dimensional measurement device 20 based on the measurement data acquired by the measurement data acquisition unit 81. The work target is the haul vehicle LS including the vessel BE. The three-dimensional data of the work target indicates the three-dimensional shape of the haul vehicle LS. The three-dimensional data of the haul vehicle LS calculated by the target calculation unit 86 is output to the working equipment control unit 87 and the determination unit 91.
The target calculation unit 86 performs stereo processing based on the image data acquired by the first camera 22A and the image data acquired by the second camera 22B to measure the three-dimensional shape of the work target. The target calculation unit 86 performs stereo processing on the image data (the first image data and the second image data) to calculate distances from the stereo camera 22 to a plurality of measurement points on the surface of the work target captured in each pixel. The target calculation unit 86 calculates, for example, three-dimensional data in the vehicle body coordinate system based on the distance to each measurement point.
In the present embodiment, the target calculation unit 86 calculates parameters related to the haul vehicle LS based on the three-dimensional data of the haul vehicle LS. The parameters related to the haul vehicle LS include the position (height) Hb of the upper end BEt of the haul vehicle LS (vessel BE) based on the ground surface RS, and the distance db from the wheel loader 1 to the haul vehicle LS. The distance db from the wheel loader 1 to the haul vehicle LS is a distance between the distal end 12B of the bucket 12 and a nearest point indicating a portion of the haul vehicle LS closest to the distal end 12B of the bucket 12 in the horizontal direction, for example.
Based on the three-dimensional data of the work target calculated by the target calculation unit 86, the working equipment control unit 87 performs intervention control of controlling the motion of the working equipment 10 to load the excavated object into the work target. In the present embodiment, the working equipment control unit 87 controls the motion of the working equipment 10 to load the excavated object onto the vessel BE based on the calculated three-dimensional data of the haul vehicle LS. The working equipment control unit 87 controls the motion of the working equipment 10 to load the excavated object onto the vessel BE based on the height data indicating the height Hb of the upper end BEt of the vessel BE and the distance data indicating the distance db from the wheel loader 1 to the haul vehicle LS.
The control of the operation of the working equipment 10 by the working equipment control unit 87 includes control of the operation of at least one of the boom cylinder 13 and the bucket cylinder 14. More specifically, the working equipment control unit 87 outputs a control signal to the boom control valve to control the flow rate and direction of the hydraulic oil to be supplied to the boom cylinder 13, thereby controlling the raising/lowering motion of the boom 11. The working equipment control unit 87 outputs a control signal to the bucket control valve to control the flow rate and direction of the hydraulic oil to be supplied to the bucket cylinder 14, thereby controlling the raising/lowering motion of the bucket 12.
The intervention control includes control of raising the bucket 12 of the working equipment 10 toward the loading target. The intervention control may include control of raising the bucket 12 of the working equipment 10 toward the loading target, control of loading the excavated object in the bucket 12 of the working equipment 10 to the loading destination, and control of lowering the bucket 12 of the working equipment 10.
The operation of the working equipment 10 to be a target of the intervention control will be described with reference to FIGS. 5 to 7 . In FIGS. 5 to 7 , a predetermined range of the angle of the working equipment 10 is indicated by A1. A measurement range of the three-dimensional measurement device 20, in other words, an imaging range of the stereo camera 22 (a visual field region of the optical system of the stereo camera 22) is indicated by A2. In the measurement range A2 of the three-dimensional measurement device 20, not only the haul vehicle LS but also the working equipment 10, the ground surface RS, or an object around the haul vehicle LS exist as a work target, for example. A detection range of the detection device 25, in other words, a scanning range of the non-contact sensor 26 (scanning region of the non-contact sensor 26) is indicated by A3. In the scanning region A3 of the non-contact sensor 26, which is the detection range of the detection device 25, not only the haul vehicle LS but also the working equipment 10 is present as a work target, for example. The non-contact sensor 26 is installed at a position different from the position of the three-dimensional measurement device 20. For example, the non-contact sensor 26 is installed on a front axle.
FIG. 5 is a diagram illustrating operation of raising the bucket 12. In FIG. 5 , the bucket 12 of the working equipment 10 is raised to the vessel BE of the haul vehicle LS. In a case where the bucket 12 is raised, the vessel BE needs to be present in the vicinity of the bucket 12.
FIG. 6 is a diagram illustrating operation of loading an excavated object of the bucket 12 to a loading destination. In FIG. 6 , the excavated object in the bucket 12 of the working equipment 10 is being loaded onto the vessel BE of the haul vehicle LS. In a case where loading an excavated object in the bucket 12, the vessel BE needs to be present in the vicinity of the bucket 12.
FIG. 7 is a diagram illustrating operation of lowering the bucket 12. In FIG. 7 , the bucket 12 of the working equipment 10 is lowered from the height Hb of the vessel BE of the haul vehicle LS. In a case where the bucket 12 is lowered, the vessel BE needs to be present in the vicinity of the bucket 12.
When both the three-dimensional measurement device 20 and the detection device 25 have detected the haul vehicle LS, the working equipment control unit 87 performs intervention control of the working equipment 10. In the present embodiment, the working equipment control unit 87 performs intervention control of the working equipment 10 when both the stereo camera 22 and the non-contact sensor 26 have detected the haul vehicle LS. The working equipment control unit 87 performs intervention control based on the position of the loading target calculated by the target calculation unit 86. In a case where the loading target has not been detected by both the three-dimensional measurement device 20 and the detection device 25, the working equipment control unit 87 stops the raising of the working equipment 10. In the present embodiment, in a case where the loading target has not been detected by both the stereo camera 22 and the non-contact sensor 26, the working equipment control unit 87 stops the raising of the working equipment 10.
In a case where the loading target has not been detected by both the three-dimensional measurement device 20 and the detection device 25, the working equipment control unit 87 may stop at least one of the raising of the working equipment 10, the loading, and the lowering of the working equipment 10. In the present embodiment, in a case where the loading target has not been detected by both the stereo camera 22 and the non-contact sensor 26, the working equipment control unit 87 may stop at least one of the raising of the working equipment 10, the loading, and the lowering of the working equipment 10.
In the present embodiment, the wheel loader 1 includes a transmission control unit 88 and a travel control unit 89.
The transmission control unit 88 outputs a control signal for controlling the transmission device 30.
The travel control unit 89 controls the motion of the traveling device 4 based on the operation on the travel operation device 40 by the driver. The travel control unit 89 outputs a driving command for activating the traveling device 4. The travel control unit 89 outputs an accelerator command for activating the driving device 4A. The travel control unit 89 outputs a braking command for operating the braking device 4B. The travel control unit 89 outputs a steering command for activating the steering device 4C.
The determination unit 91 determines whether the haul vehicle LS has been detected by both the three-dimensional measurement device 20 and the detection device 25. In the present embodiment, the determination unit 91 determines whether the haul vehicle LS has been detected by both the stereo camera 22 and the non-contact sensor 26. More specifically, the determination unit 91 determines whether the three-dimensional data of the haul vehicle LS has been calculated based on the measurement data by the target calculation unit 86 and whether the haul vehicle LS has been detected based on the detection data.
FIG. 8 is a diagram illustrating an example of processing based on a determination result. In a case where both the stereo camera 22 and the non-contact sensor 26 have detected the haul vehicle LS, in other words, in a case where both the three-dimensional measurement device 20 and the detection device 25 have detected the haul vehicle LS, the intervention control is to be activated. In this case, the display of a successful detection is performed. For example, the lamp 8 may be turned to steady-on as a display indicating the successful detection.
In a case where the three-dimensional measurement device 20 has detected the haul vehicle LS and the detection device 25 has not detected the haul vehicle LS, or in a case where the three-dimensional measurement device 20 has not detected the haul vehicle LS and the detection device 25 has detected the haul vehicle LS, the intervention control is not to be activated. In this case, the display of an abnormal detection is performed. For example, the lamp 8 may be set to blink on/off as a display indicating the abnormal detection.
In a case where neither the stereo camera 22 nor the non-contact sensor 26 have detected the haul vehicle LS, in other words, in a case where neither the three-dimensional measurement device 20 nor the detection device 25 have detected the haul vehicle LS, the intervention control is not to be activated. In this case, the display of no detection is performed. For example, the lamp 8 may be set to steady-off as a display indicating the no detection.
In the present embodiment, the determination unit 91 may make the determination in a case where the boom angle of the working equipment 10 is equal to or greater than an angle threshold θ. For example, the determination may be made in a case where the working equipment 10 is equal to or greater than the angle threshold θ, that is, in a case where the angle is outside the scanning range of the non-contact sensor 26. Alternatively, the determination may be made in a case where the angle is equal to or greater than the angle threshold θ, which is greater than a predetermined range A1. In a case where the bucket 12 is located below the vessel BE of the haul vehicle LS, the radio wave emitted by the non-contact sensor 26 is reflected by the surface of the bucket 12 and does not reach the haul vehicle LS. This might lead to a failure in detecting the haul vehicle LS correctly. Therefore, the determination unit 91 performs the determination in a case where the angle of the boom of the working equipment 10 is equal to or greater than the angle threshold θ when the bucket 12 is raised.
The output control unit 92 controls to output the determination result of the determination unit 91. The output control unit 92 controls to output a warning sound from the buzzer 7 in a case where the determination unit 91 has determined that either one of the devices has been successful in detection. The output control unit 92 controls to set the lamp 8 to blink of/off in a case where the determination unit 91 has determined that either one of the devices has been successful in detection. The output control unit 92 controls to turn the lamp 8 to steady-on in a case where the determination unit 91 has determined that both devices have been successful in detection.
[Control Method]
FIG. 9 is a flowchart illustrating a method of controlling the wheel loader 1 according to the present embodiment. As an example, loading work will be described. At the time of loading work, a haul vehicle detection mode is activated by the driver via an operation unit (not illustrated). In the loading work mode, the stereo camera 22 being the three-dimensional measurement device 20 and the non-contact sensor 26 being the detection device 25 detect a work target.
The haul vehicle LS is measured by the stereo camera 22 (Step S11). More specifically, the stereo camera 22 measures the front. The measurement data of the stereo camera 22 is output to the control device 80. The control device 80 uses the measurement data acquisition unit 81 to acquire image data captured by the stereo camera 22. The image data of the work target acquired by the measurement data acquisition unit 81 is output to the target calculation unit 86. The control device 80 calculates the position of the loading target of the wheel loader 1 by the target calculation unit 86 based on the measurement result of the stereo camera 220. The control device 80 proceeds to Step S12.
The haul vehicle LS is detected by the non-contact sensor 26 (Step S12). More specifically, the non-contact sensor 26 scans the front by radio waves. Detection data obtained by the non-contact sensor 26 is output to the control device 80. The control device 80 uses the detection data acquisition unit 84 to acquire detection data detected by the non-contact sensor 26. Detection data of the detection target acquired by the detection data acquisition unit 84 is output to the determination unit 91. The control device 80 proceeds to Step S13.
The control device 80 uses the determination unit 91 to determine whether the stereo camera 22 has detected the haul vehicle LS (Step S13). In a case where the determination unit 91 determines that the stereo camera 22 has detected the haul vehicle LS (Yes in Step S13), the control device 80 proceeds to Step S14. In a case where the determination unit 91 does not determine that the stereo camera 22 has detected the haul vehicle LS (No in Step S13), the control device 80 proceeds to Step S16.
The control device 80 uses the determination unit 91 to determine whether the non-contact sensor 26 has detected the haul vehicle LS (Step S14). In a case where the determination unit 91 determines that the non-contact sensor 26 has detected the haul vehicle LS (Yes in Step S14), the control device 80 proceeds to Step S15. In a case where the determination unit 91 does not determine that the non-contact sensor 26 has detected the haul vehicle LS (No in Step S14), the control device 80 proceeds to Step S16.
The control device 80 activates the intervention control (Step S15). The control device 80 uses the working equipment control unit 87 to control the working equipment 10 based on the height Hb of the vessel BE and the distance db to the haul vehicle LS, which have been calculated by the target calculation unit 86. The control device 80 ends the processing.
As described with reference to FIG. 3 , in a state where the wheel loader 1 moves forward so as to approach the haul vehicle LS, the working equipment control unit 87 causes the boom 11 to take a raising motion while controlling the angle of the bucket 12 so that the bucket 12 is disposed above the upper end BEt of the vessel BE and the excavated object held by the bucket 12 does not spill from the bucket 12 based on the distance db to the haul vehicle LS and the height Hb of the upper end BEt of the vessel BE calculated by the target calculation unit 86. The working equipment control unit 87 sets the bucket 12 to be disposed above the vessel BE. In addition, the working equipment control unit 87 controls the working equipment 10 so that the bucket 12 takes a dumping motion. In this manner, the excavated object is discharged from the bucket 12 and loaded onto the vessel BE.
At the time of execution of Step S15, the control device 80 may cause the output control unit 92 to turn the lamp 8 to steady-on.
The control device 80 deactivates the intervention control (Step S16). The control device 80 stops the control of the working equipment 10 performed by the working equipment control unit 87. At the time of execution of Step S16, the control device 80 may cause the output control unit 92 to output a sound from the buzzer 7. The control device 80 may set the lamp 8 to blink on/off by the output control unit 92. The control device 80 ends the processing.
FIG. 10 is a flowchart illustrating a method of detecting a work target by the non-contact sensor 26 being the detection device 25. The processing of FIG. 10 is executed after execution of the processing of Step S12 of FIG. 9 . At the start of the processing of FIG. 10 , the counts of the time of presence of detection and the time of absence of detection are reset.
In the loading work mode, the angle sensor 50 detects the angle of the working equipment 10. The angle of the working equipment 10 includes the angle of the boom 11 detected by the boom angle sensor 51 and the angle of the bucket 12 detected by the bucket angle sensor 52. The angle data indicating the angle of the working equipment 10 is output to the position data calculation unit 83.
The control device 80 determines whether the boom angle is greater than or equal to an angle threshold θ (Step S21). In the control device 80, the position data calculation unit 83 calculates the position data of the working equipment 10 based on the angle data of the working equipment 10 and the working equipment data of the working equipment 10 stored in the storage unit 82. When having determined that the boom angle is greater than or equal to the angle threshold θ (Yes in Step S21), the control device 80 proceeds to Step S22. When not having determined that the boom angle is greater than or equal to the angle threshold θ (No in Step S21), the control device 80 executes the processing of Step S21 again.
The control device 80 determines whether there is a detection point within the region (Step S22). In a case where the detection data acquisition unit 84 determines that there is a detection point within the scanning region A3 of the non-contact sensor 26 (Yes in Step S22), the control device 80 proceeds to Step S23. In a case where the detection data acquisition unit 84 does not determine that there is a detection point within the scanning region A3 of the non-contact sensor 26 (No in Step S22), the control device 80 proceeds to Step S26.
The control device 80 updates the time of presence of detection by the detection data acquisition unit 84 (Step S23). The control device 80 proceeds to Step S24.
The control device 80 determines whether the presence of a detection point continues for a predetermined time or more (Step S24). The predetermined time is any chosen time. In a case where the detection data acquisition unit 84 determines that the presence of the detection point has continued for the predetermined time or more (Yes in Step S24), the control device 80 proceeds to Step S25. In a case where the detection data acquisition unit 84 does not determine that the presence of the detection point has continued for the predetermined time or more (No in Step S24), the control device 80 executes the processing of Step S21 again.
The control device 80 confirms the presence of detection by the detection data acquisition unit 84 (Step S25). The control device 80 determines that the haul vehicle LS has been detected by the detection data acquisition unit 84. The control device 80 ends the processing.
In a case where it is not determined that there is a detection point within the scanning region A3 of the non-contact sensor 26 (No in Step S22), the control device 80 updates the time of absence of detection by the detection data acquisition unit 84 (Step S26). The control device 80 proceeds to Step S27.
The control device 80 determines whether the absence of the detection point has continued for a predetermined time or more (Step S27). In a case where the control device 80 determines by the detection data acquisition unit 84 that the absence of the detection point has continued for the predetermined time or more (Yes in Step S27), the processing proceeds to Step S28. In a case where the detection data acquisition unit 84 does not determine that the absence of the detection point has continued for the predetermined time or more (No in Step S27), the control device 80 executes the processing of Step S21 again.
The control device 80 confirms the absence of detection by the detection data acquisition unit 84 (Step S28). The control device 80 determines that the haul vehicle LS has not been detected by the detection data acquisition unit 84. The control device 80 ends the processing.
[Computer System]
FIG. 11 is a block diagram illustrating an example of a computer system 1000. The control device 80 described above is constituted with a computer system 1000. The computer system 1000 includes: a processor 1001 such as a central processing unit (CPU); main memory 1002 including non-volatile memory such as read only memory (ROM) and volatile memory such as random access memory (RAM); storage 1003; and an interface 1004 including an input/output circuit. The functions of the control device 80 described above are stored in the storage 1003 as a program. The processor 1001 reads the program from the storage 1003, expands the program to the main memory 1002, and executes the above-described processes according to the program. The program may be delivered to the computer system 1000 via a network.
[Effects]
As described above, according to the present embodiment, the intervention control of the wheel loader 1 is performed when the haul vehicle LS has been detected by both the three-dimensional measurement device 20 and the detection device 25. According to the present embodiment, it is possible to determine the presence or absence of the haul vehicle LS with higher accuracy when performing the intervention control of the wheel loader 1.
In the present embodiment, when one of the three-dimensional measurement device 20 and the detection device 25 has detected the haul vehicle LS and the other of the devices has not detected the haul vehicle LS, the intervention control is stopped. According to the present embodiment, in a case where there is a possibility of absence of the haul vehicle LS, the intervention control can be stopped.
In the present embodiment, when one of the three-dimensional measurement device 20 and the detection device 25 has detected the haul vehicle LS and the other has not detected the haul vehicle LS, a warning is output. According to the present embodiment, in a case where there is a possibility of absence of the haul vehicle LS, a warning can be output to notify the driver.
In the present embodiment, the intervention control is performed based on the calculated position of the loading target. According to the present embodiment, intervention control can be performed with high accuracy. In the present embodiment, when the boom angle of the wheel loader 1 is equal to or greater than the angle threshold θ and the haul vehicle LS has been detected by both the three-dimensional measurement device 20 and the detection device 25, the intervention control is performed. According to the present embodiment, detection can be performed when the haul vehicle LS can be detected by the detection device 25.
In the present embodiment, when one of the three-dimensional measurement device 20 and the detection device 25 has detected the haul vehicle LS and the other of the devices has not detected the haul vehicle LS, the raising of the bucket 12 can be stopped.
In the present embodiment, when one of the three-dimensional measurement device 20 and the detection device 25 has detected the haul vehicle LS and the other of the devices has not detected the haul vehicle LS, it is possible to stop at least one of the raising of the bucket 12, the loading, and the lowering of the bucket 12.
In the present embodiment, the detection device 25 is installed at a position different from the position of the three-dimensional measurement device 20. According to the present embodiment, the presence or absence of the haul vehicle LS can be determined with higher accuracy.

Other Embodiments

In each of the above-described embodiments, the three-dimensional measurement device 20 is not limited to the stereo camera 22, and may be a laser scanner, for example.
The detection device 25 is not limited to a radar device that scans with millimeter waves. The non-contact sensor 26 may include a laser scanner device that detects an object by scanning the surroundings of the wheel loader 1 with laser light. The non-contact sensor 26 may include an ultrasonic sensor device that detects an object by scanning the surroundings of the wheel loader 1 with ultrasonic waves.
In the above, when one of the three-dimensional measurement device 20 and the detection device 25 has detected the haul vehicle LS and the other of the devices has not detected the haul vehicle LS, the intervention control is stopped. However, the operation is not limited to this. For example, in a case where the working equipment 10 is raised by the intervention control, the operation may be continued in a state where the warning is output.
The work site where the wheel loader 1 performs work may be a mining site, a building site, or a construction site.
The wheel loader 1 may be used for snow removal work, work in an agricultural and livestock industry, or work in forestry.
In the above-described embodiment, the bucket 12 may have a plurality of blades or may have a straight blade edge.
The work member connected to the distal end of the boom 11 need not be the bucket 12, and may be a snow plow or a snow bucket used for snow removal work, a bale grab or a fork used for work of agriculture and livestock industry, or a fork or a bucket used for work of forestry.
Instead of the lamp 8, a monitor (not illustrated) set in the wheel loader 1 can be used to display the determination result. The wheel loader 1 need not be equipped with all of the buzzer 7, the lamp 8, and the monitor. It is sufficient that the wheel loader 1 include one or more of the devices. Alternatively, the buzzer 7, the lamp 8, and the monitor may be provided outside the wheel loader 1.
In the control system 200 according to the above-described embodiment, a part of the configuration constituting the control system 200 may be mounted inside the work machine 1, and the other configurations may be provided outside the work machine 1. In addition, the control system 200 according to the above-described embodiment is supposed to include the working equipment 10, the three-dimensional measurement device 20, the angle sensor 50, the travel operation device 40, the buzzer 7, the lamp 8, and the control device 80. However, the configuration is not limited thereto, and some configurations may be omitted. As an example, it is possible to provide the control system 200 not including the buzzer 7 or the lamp 8.
The control device 80 according to the above-described embodiment may be constituted with a single computer, or configurations of the control device 80 may be divided into a plurality of computers, and the plurality of computers may function as the control device 80 in cooperation with each other.
The work machine 1 is not limited to the wheel loader, and the control device 80 and the control method described in the above embodiment are applicable to a work machine having working equipment such as an excavator or a bulldozer.

REFERENCE SIGNS LIST

- 1 WHEEL LOADER (WORK MACHINE)
- 2 VEHICLE BODY
- 2F VEHICLE BODY FRONT PORTION
- 2R VEHICLE BODY REAR PORTION
- 3 CAB
- 4 TRAVELING DEVICE
- 4A DRIVING DEVICE
- 4B BRAKING DEVICE
- 4C STEERING DEVICE
- 5 WHEEL
- 5F FRONT WHEEL
- 5R REAR WHEEL
- 6 TIRE
- 6F FRONT TIRE
- 6R REAR TIRE
- 7 BUZZER (OUTPUT UNIT)
- 8 LAMP (OUTPUT UNIT)
- 9 JOINT MECHANISM
- 10 WORKING EQUIPMENT
- 11 BOOM
- 12 BUCKET
- 12B DISTAL END
- 13 BOOM CYLINDER
- 14 BUCKET CYLINDER
- 15 BELL CRANK
- 16 LINK
- 20 THREE-DIMENSIONAL MEASUREMENT DEVICE
- 22 STEREO CAMERA
- 22A FIRST CAMERA
- 22B SECOND CAMERA
- 25 DETECTION DEVICE
- 26 NON-CONTACT SENSOR
- 30 TRANSMISSION DEVICE
- 40 TRAVEL OPERATION DEVICE
- 50 ANGLE SENSOR
- 51 BOOM ANGLE SENSOR
- 52 BUCKET ANGLE SENSOR
- 80 CONTROL DEVICE
- 81 MEASUREMENT DATA ACQUISITION UNIT
- 82 STORAGE UNIT
- 83 POSITION DATA CALCULATION UNIT
- 84 DETECTION DATA ACQUISITION UNIT
- 86 TARGET CALCULATION UNIT
- 87 WORKING EQUIPMENT CONTROL UNIT
- 88 TRANSMISSION CONTROL UNIT
- 89 TRAVEL CONTROL UNIT
- 91 DETERMINATION UNIT
- 92 OUTPUT CONTROL UNIT
- 100 IMAGE DATA
- 200 ABNORMALITY DETERMINATION SYSTEM
- BE VESSEL (LOADING TARGET)
- DS NATURAL HILL (EXCAVATION TARGET)
- FX ROTATION AXIS
- LS HAUL VEHICLE
- RX ROTATION AXIS
- RS GROUND SURFACE

Claims

1. A control system of a work machine, the system comprising:

a three-dimensional measurement device that measures a work target of the work machine;

a detection device that detects the work target; and

an intervention control unit that performs intervention control of the work machine when both the three-dimensional measurement device and the detection device detects the work target.

2. The control system of the work machine according to claim 1,

wherein the intervention control unit stops the intervention control when one of the three-dimensional measurement device and the detection device detects the work target and the other of the three-dimensional measurement device and the detection device does not detect the work target.

3. The control system of the work machine according to claim 1, further comprising

an output unit that outputs a warning when one of the three-dimensional measurement device and the detection device detects the work target and the other of the three-dimensional measurement device and the detection device does not detect the work target.

4. The control system of the work machine according to claim 1, further comprising

a target calculation unit that calculates a position of a loading target of the work machine based on a measurement result of the three-dimensional measurement device,

wherein the intervention control unit performs intervention control based on the position of the loading target calculated by the target calculation unit.

5. The control system of the work machine according to claim 1,

wherein the intervention control unit performs intervention control when an angle of working equipment of the work machine is equal to or greater than an angle threshold and the work target is detected by both the three-dimensional measurement device and the detection device.

6. The control system of the work machine according to claim 1,

wherein the intervention control includes control of raising working equipment of the work machine toward a loading target of the work machine, and

the intervention control unit stops raising of the working equipment when one of the three-dimensional measurement device and the detection device detects the work target and the other of the three-dimensional measurement device and the detection device does not detect the work target.

7. The control system of the work machine according to claim 1,

wherein the intervention control includes control of raising working equipment of the work machine toward a loading target of the work machine, control of loading an excavated object of the working equipment onto the loading target, and control of lowering the working equipment, and

the intervention control unit stops at least one of the raising of the working equipment, the loading of the working equipment, and the lowering of the working equipment when one of the three-dimensional measurement device and the detection device detects the work target and the other of the three-dimensional measurement device and the detection device does not detect the work target.

8. The control system of the work machine according to claim 1,

wherein the detection device is installed at a position different from a position of the three-dimensional measurement device.

9. The control system of the work machine according to claim 1,

wherein the work machine is a wheel loader,

the detection device is installed on a front axle, and

the intervention control unit performs intervention control when the work target is detected by both the three-dimensional measurement device and the detection device in a case where an angle of working equipment is equal to or greater than an angle threshold.

10. A method of controlling a work machine comprising

performing intervention control of the work machine when both a three-dimensional measurement device that measures a work target of the work machine and a detection device that detects the work target detect the work target.

11. The method of controlling the work machine according to claim 10,

wherein intervention control is stopped when one of the three-dimensional measurement device and the detection device detects the work target and the other of the three-dimensional measurement device and the detection device does not detect the work target.

12. The method of controlling the work machine according to claim 10,

wherein a warning is output when one of the three-dimensional measurement device and the detection device detects the work target and the other of the three-dimensional measurement device and the detection device does not detect the work target.

13. The method of controlling the work machine according to claim 10, further comprising:

calculating a position of a loading target of the work machine based on a measurement result of the three-dimensional measurement device, and

performing intervention control based on the position of the loading target obtained by the calculation.

14. The method of controlling the work machine according to claim 10,

wherein intervention control is performed when an angle of working equipment of the work machine is equal to or greater than an angle threshold and the work target is detected by both the three-dimensional measurement device and the detection device.

15. The method of controlling the work machine according to claim 10,

the raising of the working equipment is to be stopped when one of the three-dimensional measurement device and the detection device detects the work target and the other of the three-dimensional measurement device and the detection device does not detect the work target.

16. The method of controlling the work machine according to claim 10,

at least any one of the raising of the working equipment, the loading of the working equipment, and the lowering of the working equipment is to be stopped when one of the three-dimensional measurement device and the detection device detects the work target and the other of the three-dimensional measurement device and the detection device does not detect the work target.