JP2024004776A - Work machine - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a work machine capable of accurately detecting the position of a tray even for a transport vehicle on which there is no data.

SOLUTION: A work machine for loading objects to be transported onto a tray of a transport vehicle is provided with: a traveling body; a superstructure rotatably arranged on an upper part of the traveling body; a front work machine attached on the superstructure; a measurement device for measuring objects surrounding the work machine to output a group of points indicating the measurement results; and a processing device for carrying out a detection process of the tray from the group of points output by the measurement device, wherein the processing device samples two faces opposite to each other in a horizontal direction from the group of points output by the measurement device and determines whether or not a distance between the sampled two faces is longer than a predetermined set distance. When the distance between the sampled faces is longer than the predetermined set distance, it detects the tray having the two faces as right and left walls and outputs a position of the tray upon detection of the tray.

SELECTED DRAWING: Figure 14

COPYRIGHT: (C)2024,JPO&INPIT

Description

The present invention relates to working machines such as hydraulic excavators.

Work machines such as hydraulic excavators that have multi-jointed front work machines are known. This type of working machine is often used for loading work in which excavated materials such as earth and sand are loaded onto a transport vehicle such as a dump truck. Loading operations include a transportation operation in which the revolving body rotates relative to the traveling body and transports the object to a transportation vehicle such as a dump truck, and a discharge operation in which the bucket is clouded and the transported object is discharged to the transportation vehicle (e.g., release operation). Earth movements) are included.

When carrying out loading work, if the revolving structure turns with the height of the front work equipment (for example, the height of the tip of the bucket) lower than the tray of the transport vehicle, the front work equipment may interfere with the transport vehicle during the transport process. there's a possibility that. Therefore, there is a need for a function that supports an operator's operation of a working machine during loading work, and for a technology that allows a working machine to automatically perform the loading work. In order to automatically or semi-automatically carry out the loading work of a working machine, it is necessary to accurately detect the position and angle of the transport vehicle with respect to the working machine.

BACKGROUND ART As a conventional technique for detecting a transport vehicle to be loaded in a loading operation of a work machine, there is a technique described in Patent Document 1, for example. Patent Document 1 discloses an image processing system that identifies an area including the transport vehicle from an image in which the transport vehicle is captured, and identifies at least one surface of the transport vehicle from the area including the transport vehicle.

Japanese Patent Application Publication No. 2020-126363

The image recognition system described in Patent Document 1 uses machine learning, and based on data obtained by machine learning, the position and angle of a transport vehicle are detected from an image taken by a camera attached to a working machine. However, in systems that use machine learning, if the site or the transport vehicle has not yet been trained, the accuracy and validity of the detection results of the position and angle of the transport vehicle will decrease. Therefore, when detecting a transport vehicle at a new site or when detecting a new transport vehicle, learning data for those new sites and transport vehicles must be collected in order to ensure the validity and accuracy of the detection results. Must be.

An object of the present invention is to provide a working machine that can accurately detect the position of a tray even in a transport vehicle without data.

In order to achieve the above object, the present invention provides a working machine for loading cargo onto a tray of a transport vehicle, which includes a traveling body, a revolving body rotatably provided on the upper part of the traveling body, and a revolving body attached to the revolving body. a measuring device that measures objects around the working machine and outputs a point cloud indicating the measurement results, and performs detection processing of the tray from the point cloud output by the measuring device. a processing device, the processing device extracts two surfaces facing each other in the horizontal direction from the point cloud output by the measurement device, and determines whether the distance between the two extracted surfaces is longer than a predetermined set distance. If it is determined that the distance between the surfaces is longer than the set distance, the tray with the two surfaces as the left and right walls is detected, and when the tray is detected, the position of the tray is output. To provide a working machine characterized by:

According to the present invention, it is possible to accurately detect the position of a tray even in a transportation vehicle for which there is no data.

A side view of a hydraulic excavator that is an example of a working machine according to the present invention A block diagram showing the main parts of the hydraulic system and control system included in the hydraulic excavator shown in Figure 1, along with related components. A plan view showing an example of a hydraulic excavator's work Side view showing an example of work of a hydraulic excavator An example of a status data table A side view showing a reference coordinate system together with a side view of a working machine according to a first embodiment of the present invention A plan view showing a reference coordinate system together with a side view of a working machine according to a first embodiment of the invention. Functional block diagram regarding tray detection processing of a transport vehicle by a processing device provided in a working machine according to a first embodiment of the present invention Side view showing the reference coordinate system of the transport vehicle Plan view showing the reference coordinate system of the transport vehicle A flowchart showing an example of a series of operations including tray detection processing of the work machine according to the first embodiment of the present invention Flowchart showing an example of the procedure of the situation determination process (FIG. 8) Flowchart showing an example of the procedure of transportation vehicle detection processing (FIG. 8) Flowchart showing an example of tray detection processing procedure Schematic diagram showing an example of a cross section of a tray of a transport vehicle Schematic diagram showing an example of a cross section of a tray of a transport vehicle Schematic diagram showing an example of a cross section of a tray of a transport vehicle Schematic diagram showing an example of a cross section of a tray of a transport vehicle Flowchart showing an example of the procedure of detection result correction processing (FIG. 8) An explanatory diagram of an example of tray detection data correction An explanatory diagram of an example of tray detection data correction An explanatory diagram of an example of tray length correction Diagram showing an example of displaying tray detection results Functional blocks related to tray detection processing of a transport vehicle by a processing device provided in a working machine according to a second embodiment of the present invention

Embodiments of the present invention will be described below using the drawings.

(First embodiment)
1. Working Machine FIG. 1 is a side view of a hydraulic excavator which is an example of a working machine according to the present invention, and FIG. 2 is a block diagram showing the main parts of the hydraulic system and control system provided in the hydraulic excavator shown in FIG. 1 along with related components. It is a diagram. In this specification, the left direction in FIG. 1 is defined as the front of the hydraulic excavator (strictly speaking, the rotating structure 7). Furthermore, although a large hydraulic excavator is illustrated as an example in FIG. 1, the present invention is also applicable to medium-sized or smaller hydraulic excavators, or other types of working machines having front working machines.

In Fig. 1, a hydraulic excavator 1 (work machine) performs excavation work to excavate the surface to be excavated, such as the ground, and loads materials such as excavated earth and sand onto a transport vehicle such as a dump truck (described later). It is used to perform work, etc. The loading operation includes a transportation operation in which the object scooped into the bucket 10 (described later) is transported to the transportation vehicle by the rotation of the revolving body 7 (described later) with respect to the traveling body 5 (described later), and the transportation operation from the bucket 10 to the transportation vehicle. This includes a discharge operation for discharging the conveyed object. The hydraulic excavator 1 includes a vehicle body 3 and an articulated front working machine 2 (work arm) mounted on the vehicle body 3.

The vehicle body 3 includes a traveling body 5 and a rotating body 7. The traveling body 5 is a lower structure of the hydraulic excavator 1, and includes crawler-type left and right traveling devices 5a driven by a traveling motor 4 (hydraulic motor). The revolving body 7 is rotatably attached to the upper part of the frame of the traveling body 5 via a rotating device (not shown), and is driven by a swing motor 6 (hydraulic motor) of the rotating device to rotate the rotating body 5 with respect to the traveling body 5. rotate. The revolving body 7 is equipped with a drive system for the hydraulic excavator 1 and an operator's cab 20, including a prime mover and the like.

The front working machine 2 is an articulated working device, is attached to the front part of the revolving body 7, and rotates with the revolving body 7 relative to the traveling body 5. This front working machine 2 includes a boom 8, an arm 9, and a bucket 10. The boom 8 is rotatably connected to the front part of the rotating body 7 via a boom pin 8a (FIG. 6), and is driven in the vertical direction by a boom cylinder 11. The arm 9 is rotatably connected to the tip of the boom 8 via an arm pin 9a, and is driven by an arm cylinder 12 in the cloud direction and the dump direction. The bucket 10 is rotatably connected to the tip of the arm 9 via a bucket pin 10a and a bucket link 16, and is driven by a bucket cylinder 13 in the cloud direction and the dump direction.

A boom angle sensor 14 (FIG. 6) that detects the angle of the boom 8 with respect to the rotating body 7 is attached to the boom pin 8a. An arm angle sensor 15 that detects the angle of the arm 9 with respect to the boom 8 is attached to the arm pin 9a. A bucket angle sensor 17 is attached to the bucket link 16 to detect the angle of the bucket 10 with respect to the arm 9. Note that the respective angles of the boom 8, arm 9, and bucket 10 are determined by detecting the respective angles of the boom 8, arm 9, and bucket 10 with respect to a reference plane (for example, a horizontal plane) using an inertial measurement unit (IMU), and converting them from these detected values. The information may be obtained by Further, each stroke of the boom cylinder 11, arm cylinder 12, and bucket cylinder 13 is detected by a stroke sensor, and the respective angles of the boom 8, arm 9, and bucket 10 are obtained by converting these detected values. You can also do it.

In addition, a tilt angle sensor (not shown) is attached to the revolving body 7 to detect the tilt angle of the vehicle body 3 with respect to a reference plane such as a horizontal plane. A turning angle sensor 19 is attached to a turning device that connects the traveling body 5 and the rotating body 7 to detect a turning angle that is a relative angle of the rotating body 7 with respect to the traveling body 5. Further, an angular velocity sensor (not shown) for detecting the angular velocity of the rotating body 7 is attached to the rotating body 7 .

In this embodiment, the boom angle sensor 14, the arm angle sensor 15, the bucket angle sensor 17, the tilt angle sensor (not shown), the swing angle sensor 19, and the angular velocity sensor (not shown) are collectively referred to as the attitude detection sensor 53 (not shown). 2).

Moreover, an operating device for instructing the operation of the traveling body 5, the rotating body 7, and the front working machine 2 is arranged inside the operator's cab 20 provided in the rotating body 7 (FIG. 2). The operating device includes, for example, a right travel lever 23a, a left travel lever 23b, a right operation lever 22a, and a left operation lever 22b. The right travel lever 23a is used to operate the travel motor 4 of the right travel device. The left travel lever 23b is used to operate the left travel motor 4. The right operating lever 22a is commonly used to operate the boom cylinder 11 and the bucket cylinder 13. The left operating lever 22b is commonly used to operate the arm cylinder 12 and the swing motor 6. In this specification, when the subscript is omitted and the operating lever 22 is written, it means the right operating lever 22a and the left operating lever 22b. Similarly, when describing the operation lever 23, it means the right travel lever 23a and the left travel lever 23b. The operating levers 22 and 23 are of an electric lever type, and sensors 52a to 52f (e.g., rotary encoder, potentiometer) detect the amount and direction in which the lever is operated by the operator, and output an operation signal according to the amount and direction of operation. Output.

Furthermore, a measuring device 70 is installed on the revolving body 7 to measure objects around the hydraulic excavator 1 and output a point group. This measuring device 70 is an external measuring device that measures the distance (depth) to objects existing around the hydraulic excavator 1. The measuring device 70 may be any device as long as it can output point cloud data, and for example, LiDAR (Light Detection And Ranging) or a stereo camera can be used. One or more measuring devices 70 are attached to the hydraulic excavator 1 . A suitable arrangement of the measuring device 70 will be described later.

2. Control System As shown in FIG. 2, the drive system of the hydraulic excavator 1 includes a prime mover 103, a hydraulic pump 102, a pilot pump 104, a control device 40, electromagnetic proportional valves 47a to 47l, a flow rate control valve 101, and a processing device (information processing device). equipment) 54, etc.

The prime mover 103 is, for example, an engine (or an electric motor), and drives the hydraulic pump 102 and the pilot pump 104. The control device 40 controls the operations of the front working machine 2, the traveling body 5, and the rotating body 7 according to operation signals inputted from the operation levers 22 and 23. Specifically, the control device 40 calculates a command signal based on the operation signals input from the sensors 52a to 52f in response to the operation of the control levers 22 and 23 by the operator, and outputs the command signal to the electromagnetic proportional valves 47a to 47l. The electromagnetic proportional valves 47a to 47l operate in response to command signals from the control device 40, reduce the pressure of pressure oil supplied from the pilot pump 104 via the pilot line 100, and output pilot pressure to the flow control valve 101. .

Note that the electromagnetic proportional valves 47a and 47b output pilot pressure related to the operation of the swing motor 6. The electromagnetic proportional valves 47c and 47d output pilot pressure related to the operation of the arm cylinder 12. The electromagnetic proportional valves 47e and 47f output pilot pressure related to the operation of the boom cylinder 11. The electromagnetic proportional valves 47g and 47h output pilot pressure related to the operation of the bucket cylinder 13. The electromagnetic proportional valves 47i and 47j output pilot pressure for operating the right travel motor 4. The electromagnetic proportional valves 47k and 47l output pilot pressure for operating the left travel motor 4.

The flow control valve 101 is equipped with direction control valves (not shown) corresponding to the swing motor 6, arm cylinder 12, boom cylinder 11, bucket cylinder 13, and left and right travel motors 4. Each directional control valve operates by receiving pilot pressure input from a corresponding one of the electromagnetic proportional valves 47a to 47l into a pressure receiving chamber, and controls the direction and flow rate of pressure oil supplied from the hydraulic pump 102. Supplies the corresponding hydraulic actuator. Thereby, the swing motor 6, the arm cylinder 12, the boom cylinder 11, the bucket cylinder 13, and the left and right travel motors 4 operate according to the operation of the operating levers 22 and 23 by the operator. As a result, the position and angle of the bucket 10 change, the revolving body 7 turns, and the traveling body 5 travels.

The processing device 54 is a computer, and has a function of executing tray detection processing (described later) based on the point cloud output by the measurement device 70. The processing device 54 includes a ROM (Read Only Memory) 71, a RAM (Random Access Memory) 72, a CPU (Central Processing Unit) 73, an external I/F (Interface) 74, and the like. The ROM 71, RAM 72, CPU 73, and external I/F 74 are connected via a bus 75. Further, a control device 40 , a monitor 55 , a measuring device 70 , an attitude detection sensor 53 , and a storage device 57 (for example, a hard disk drive or a large-capacity flash memory) are connected to the processing device 54 via an external I/F 74 . The monitor 55 is arranged inside the driver's cab 20 at a position where it can be easily seen by an operator sitting in the driver's seat. The functions of the processing device 54 will be described later.

3. Example of Work FIGS. 3 and 4 are a plan view and a side view showing an example of work of the hydraulic excavator 1. 3 and 4 show a loading operation in which earth and sand are excavated with the hydraulic excavator 1, and the excavated earth and sand is transported and loaded onto the tray T of the transport vehicle 200. For example, the transport vehicle 200 is placed at a position where the hydraulic excavator 1 can load objects (for example, a position within an area within the maximum turning radius from the turning center line 120 of the turning structure 7 and greater than or equal to the rear end turning radius of the turning structure 7). stops. At this time, in the hydraulic excavator 1 , the processing device 54 executes a tray detection process to detect the position and angle of the tray T of the transport vehicle 200 based on the output of the measuring device 70 . Based on the result of this tray detection process, the control device 40 of the hydraulic excavator 1 controls the hydraulic excavator 1 (for example, the front Controls the operation of work equipment 2). The results of the tray detection process can be widely applied not only to loading operations but also to interference prevention control between the tray T and the front working machine 2. For example, when moving the hydraulic excavator 1 to the next excavation point after loading operations. It can also be applied to avoid interference between the tray T and the hydraulic excavator 1.

4. Layout of Measuring Device When the work configuration illustrated in FIGS. 3 and 4 is assumed, a preferred example is to install the measuring device 70 so that it can acquire a point cloud in the area on the left side of the rotating structure 7 during excavation, for example. be. However, the installation position of the measuring device 70 is not limited to this example, and the measuring device 70 can be installed at a position where the front of the rotating body 7 can be measured, for example.

Furthermore, in the tray detection process, two sides of the tray T on the transport vehicle 200, on the left and right sides, are extracted based on the point cloud data obtained from the measuring device 70. In general, the tray T of the transport vehicle 200 has inner wall surfaces on both left and right sides formed of flat surfaces or smooth curved surfaces, so the tray T can be detected by extracting these two surfaces. In this case, it is necessary to install the measuring device 70 so that the inner wall surface of the tray T of the transport vehicle 200 can be measured. In the present embodiment, with the measuring device 70 facing the transport vehicle 200 parked at a predetermined position on the side of the hydraulic excavator 1, the measuring device 70 is placed on the extension of the vehicle width of the transport vehicle 200 (toward the rear of the transport vehicle 200). (on the projection of tray T).

Further, when the transport vehicle 200 is stopped at a low position relative to the hydraulic excavator 1 as in the example of FIGS. 3 and 4, the measuring device 70 may be installed so that the transport vehicle 200 can be viewed diagonally and measured. desirable. In this embodiment, the measuring device 70 is installed at the left end of the upper part of the driver's cab 20 with its optical axis directed diagonally downward so as to measure the distance to an object present in the left area of the rotating body 7. 3 and 4, θsh is the horizontal angle of the measuring range of the measuring device 70, θsv is the vertical angle of the measuring range of the measuring device 70, and Lsr is the distance of the measuring range of the measuring device 70 from the hydraulic excavator 1. represents.

5. Situation Data Table FIG. 5 is an example of a situation data table. The situation data table 800 shown in the figure is a data table that summarizes the current situation (transition state) related to the tray detection process and the processing contents for each situation, and is defined in advance and stored in the ROM 71, for example. In this embodiment, there are six situations related to tray detection processing: "point cloud not acquired", "transport vehicle not detected", "transport vehicle detected", "tray detected", "turning", and "traveling". Classified into For each of these states, "applicable conditions,""detection process execution capability,""detection result output capability," and "detection result management method" are defined. The current situation in the situation data table 800 is determined by a transport vehicle detection process 83 and a situation determination process 84 by the processing device 54 as described later, and is temporarily stored in a predetermined storage area of the RAM 72, for example. Explain each situation.

5-1. "Point cloud not acquired"
Point cloud not acquired is a situation where the point cloud data of the measuring device 70 is not input to the processing device 54 for a certain period of time or more due to communication conditions or the like. An example of the point cloud not being acquired may be a case where the point cloud data cannot be acquired due to a problem with the network or equipment. Under this situation, tray detection processing by the processing device 54 is not executed. During this time, the tray detection results are not output from the processing device 54 to the control device 40. Furthermore, the processing device 54 erases data held in the buffer (for example, the RAM 72) regarding the tray detection process so that the tray detection results are not output to the control device 40.

5-2. "Transportation vehicle not detected"
Transport vehicle not detected is a situation where the point cloud data from the measuring device 70 is input, but the processing device 54 does not detect anything that is estimated to be an object (excluding the ground) from the point cloud around the hydraulic excavator 1. be. Under this situation, the tray detection process by the processing device 54 is continuously executed. However, since no object is detected, the tray detection result is not output from the processing device 54 to the control device 40 (or a detection result indicating that the tray is not detected may be output). Further, as in the case where the point cloud is not detected, the processing device 54 erases the data held in the buffer (for example, the RAM 72) regarding the tray detection process.

5-3. "Transport vehicle detection"
Transport vehicle detection is performed in a situation where an object of a size equivalent to the transport vehicle 200 is detected around the hydraulic excavator 1 by the processing device 54 based on point cloud data from the measuring device 70, but the tray T is not detected. be. Typically, the transport vehicle 200 is continuously detected, but as the loading of the transport vehicle 200 progresses, the tray T is hidden by the transport vehicle, and the tray T is hidden by the transport vehicle 200. A situation is assumed in which the shape of at least one of the left and right inner wall surfaces is no longer detected. Under this situation (that is, during the loading work of the transported object), the processing device 54 continues to perform tray detection processing, and since the transport vehicle 200 remains stopped and the tray T is not moving, the latest The tray detection result is output to the control device 40 as still valid data. In other words, although tray T is not detected under the "transport vehicle detection" situation, for example, the latest tray detection result obtained and held under the "tray detection" situation immediately before transitioning to the transport vehicle detection situation is It is output to the control device 40. Furthermore, under the situation of transport vehicle detection, the latest tray detection result held in the buffer (for example, RAM 72) is continuously held.

5-4. "Tray detection"
Tray detection is a situation in which an object of a size equivalent to the transport vehicle 200 is detected around the hydraulic excavator 1 by the processing device 54 based on point cloud data from the measuring device 70, and a tray T is also detected. . Under this situation, the processing device 54 continues to execute the tray detection process and sequentially outputs the latest tray detection results obtained in real time to the control device 40. Furthermore, under the tray detection situation, the tray detection results are updated one after another, and the latest tray detection results are always held in the RAM 72.

5-5. "Turning"
During turning, the rotating body 7 of the hydraulic excavator 1 is turning at a turning speed higher than a preset speed (for example, 1 to 5 deg/sec). The turning speed is measured, for example, by an angular velocity sensor provided on the turning structure 7. In this embodiment, since the measuring device 70 is installed on the rotating body 7, while the rotating body 7 is rotating, there is a possibility that the accuracy of the tray detection result will decrease due to the decrease in the accuracy of coordinate conversion of the point cloud data in the rotating direction. There is. Therefore, regardless of whether or not the transport vehicle 200 is detected, the tray detection process by the processing device 54 is not executed. However, since the position of the transport vehicle 200 does not change before and after the rotation of the revolving body 7 during the loading operation of the transport vehicle 200, the processing device 54 detects the latest tray as in the situation of transport vehicle detection. The result is output to the control device 40 as valid data. Unless the loading work is in progress, the tray detection result data in the RAM 72 is erased, so it will not be output to the control device 40 during turning. Further, even during rotation, the latest tray detection results held in the RAM 72 are continuously held.

5-6. "Running"
While traveling, the hydraulic excavator 1 is in motion. In this embodiment, while the hydraulic excavator 1 is moving, objects to be transported are not loaded onto the transport vehicle 200, so in this embodiment, the tray detection process by the processing device 54 is not executed. Further, when the hydraulic excavator 1 moves, the relative position between the hydraulic excavator 1 and the transport vehicle 200 changes, so the control device 40 is not notified of the immediately previous tray detection result. Further, as in the case where the point cloud is not detected, the processing device 54 erases the data held in the RAM 72 regarding the tray detection process.

The traveling operation of the hydraulic excavator 1 is determined based on whether the operating amounts of the right traveling lever 23a and the left traveling lever 23b detected by the sensors 52e and 52f are equal to or greater than a preset value. However, the method for detecting the traveling motion of the hydraulic excavator 1 is not limited to this, and various methods can be provided, such as measuring the vehicle speed of the hydraulic excavator 1 with a speedometer, or measuring the rotation speed of the travel motor 4 with a rotation speed sensor. It can also be detected by

6. Processing device 6-1. Overview The processing device 54 executes a tray detection process of detecting the tray T of the transport vehicle 200 from the point cloud output by the measurement device 70. In the tray detection process, the processing device 54 first extracts two horizontally opposing surfaces from the point cloud output by the measuring device 70, and determines whether the distance between the two extracted surfaces is longer than a preset distance. do. As a result, if the distance between the two surfaces is longer than the set distance, the processing device 54 determines that the two extracted surfaces are the left and right wall surfaces of the tray T, and the processing device 54 determines that the two extracted surfaces are the left and right wall surfaces of the tray T. T is detected, and the current position of the tray T is output to the control device 40. The set distance to be compared with the extracted distance between the two surfaces is at least larger than the width in the left-right direction of the bucket 10 of the front working machine 2, and is, for example, the general vehicle width of the vehicle assumed as the transport vehicle 200 or the like. is set slightly smaller than .

Preferably, this tray detection process can be executed on the premise that the transport vehicle 200 has been detected. In this case, the processing device 54 detects objects larger than a preset transport vehicle size or a circular shape corresponding to a running wheel from the point cloud output by the measurement device 70 prior to the tray detection process. When a corresponding object is detected, the object is detected as the transport vehicle 200. For example, when the site is a mine, the width of the dump truck used as the transport vehicle 200 is generally small and is about 7 m, so the size equivalent to the transport vehicle to be compared with the extracted object is a slightly smaller value ( For example, 5 to 6 m) is assumed. After detecting the transport vehicle 200 in this way, the processing device 54 extracts the two surfaces facing each other in the horizontal direction from the point group of the object detected as the transport vehicle 200 as part of the tray detection process.

When the point cloud is not input from the measuring device 70 and when the transport vehicle 200 is not detected, the tray detection process by the processing device 54 is interrupted (not executed). In addition to this, preferably, the tray detection process by the processing device 54 can be interrupted (not executed) in a predetermined case regardless of whether or not the transport vehicle 200 is detected. In the case of this embodiment, as explained in FIG. 5 on the left, the tray detection process by the processing device 54 is not executed while the vehicle is turning or traveling. For example, the processing device 54 is programmed to not execute the tray detection process when it is determined that the rotating body 7 is rotating based on the output of the angular velocity sensor (for example, when the rotating speed is 1 deg/sec or more). be done. Further, the processing device 54 is programmed not to perform the tray detection process when it is determined that the hydraulic excavator 1 is traveling based on the output of the traveling sensors (for example, the sensors 52e and 52f).

The processing device 54 holds the latest detection data of the tray T obtained in the tray detection process in predetermined cases (for example, in each situation of transport vehicle detection, tray detection, and turning in FIG. 5). Even when the tray T is no longer detected, the processing device 54 uses the latest detection data held while the transport vehicle 200 is being detected (for example, during transport vehicle detection and turning in FIG. 5). The data is output to the control device 40 as being equivalent to data detected in real time.

Preferably, the processing device 54 corrects the tray detection result in a predetermined case during the tray detection process. As an example, the lengths in the longitudinal direction of the two sides extracted during tray detection processing are compared, and if the lengths in the longitudinal direction of the two sides are different, the length is shortened according to the detection data (length) of the longer side. The processing device 54 can be programmed so that the detected data (length) of one side is corrected. As another example, the processing device 54 compares the longitudinal length of the tray T detected in the tray detection process with a threshold value, and if the detected length of the tray T is shorter than the threshold value, the processing device 54 determines a predetermined tray length ( For example, the length of the tray T is programmed to be corrected to a value set slightly longer than a threshold value. The threshold value in this case is a value generally assumed as the front-to-back length of the tray T of the transport vehicle 200 or a preset value slightly shorter than that, and can be stored in the ROM 71, for example. When adjusting the longitudinal length of the tray T to the set tray length, the processing device 54 is programmed to correct the detected length of the tray T by extending it forward of the transport vehicle 200 based on the detection result of the transport vehicle 200. is desirable.

Furthermore, the processing device 54 displays the detection data of the tray on the monitor 55 in an overlapping manner with the point cloud from the measuring device 70 . Further, while the processing device 54 stops outputting the current position of the tray T to the control device 40, the processing device 54 displays on the monitor 55 that the current position of the tray T is not detected.

6-2. Specific Example The functions of the processing device 54 will be specifically explained.

FIG. 6 is a side view showing the reference coordinate system together with the hydraulic excavator, and FIG. 7 is a plan view thereof. First, in, for example, the ROM 71 of the processing device 54, preset vehicle body coordinates, which are a local coordinate system of the vehicle body 3 with the traveling body 5 as a reference, are stored as a reference coordinate system for specifying the positions and angles of the components of the hydraulic excavator 1. A system 400 (FIGS. 6 and 7) is stored. In the present embodiment, the vehicle body coordinate system 400 is, for example, an XYZ right-handed orthogonal coordinate system whose origin is the intersection of the turning center line 120, which is the rotational axis of the rotating body 7, and the ground contact surface (the bottom surface in contact with the ground G) of the traveling body 5. is defined as In the vehicle body coordinate system 400, the forward direction of the traveling body 5 is the positive direction of the X-axis, the direction from the traveling body 5 to the rotating body 7 along the turning center line 120 is the positive direction of the Z-axis, and the left side of the traveling body 5 is the positive direction of the Z-axis. The positive direction of the Y axis. In the vehicle body coordinate system 400, the turning angle θsw (FIG. 7) of the rotating body 7 is such that the front working device 2 faces the positive direction of the X-axis, and the center line of the front working device 2 (dotted chain line in FIG. 7) A state parallel to the X-axis of the coordinate system 400 is defined as 0 degrees.

Further, a preset xyz right-handed orthogonal coordinate system based on the measuring device 70 is the sensor coordinate system 300, and a preset X'Y'Z' right-handed orthogonal coordinate system that is the reference of the work site is the site coordinate system 500. It is stored in the ROM 71 as . In addition, Lbm in FIG. 6 is the length of the boom 8 (the distance between the centers of the boom pin 8a and the arm pin 9a), Lam is the length of the arm 9 (the distance between the centers of the arm pin 9a and the bucket pin 10a), and Lbk is the length of the bucket 10. length (distance between the center of the bucket pin 10a and the tip of the bucket).

FIG. 8 is a functional block diagram regarding tray detection processing by the processing device 54. As shown in FIG. 8, the processing device 54 executes an attitude calculation process 81, a coordinate conversion process 82, a transportation vehicle detection process 83, a situation determination process 84, and a detection result correction process 85 regarding the tray detection process. The contents of each process will be sequentially explained below.

-Posture calculation-
The posture calculation process 81 is a process of the processing device 54 that calculates the positions and angles of the components of the hydraulic excavator 1 in the vehicle body coordinate system 400 based on the detection signal of the posture detection sensor 53.

For example, based on the detection signal of the rotation angle of the boom 8 output from the boom angle sensor 14, the rotation angle θbm (FIG. 6) of the boom 8 with respect to the X axis is calculated. Further, based on the detection signal of the rotation angle of the arm 9 output from the arm angle sensor 15, the rotation angle θam (FIG. 6) of the arm 9 with respect to the boom 8 is calculated. Based on the detection signal of the rotation angle of the bucket 10 output from the bucket angle sensor 17, the rotation angle θbk (FIG. 6) of the bucket 10 with respect to the arm 9 is calculated. Based on the detection signal of the turning angle of the rotating body 7 output from the turning angle sensor 19, the turning angle θsw (FIG. 7) of the rotating body 7 with respect to the traveling body 5 is calculated.

Furthermore, based on the detection signal of the inclination angle of the vehicle body 3 output from the inclination angle sensor (not shown), the inclination angle θg (Fig. 6) of the vehicle body 3 (traveling body 5) with respect to the reference plane DP (Fig. 6) is calculated. be done. The reference plane DP is, for example, a horizontal plane perpendicular to the direction of gravity. The tilt angle θg includes a rotation angle θp (not shown) around the Y axis and a rotation angle θr (not shown) around the X axis as components. Furthermore, the turning angular velocity ωsw of the rotating body 7 is calculated based on the detection signal of the attitude detection sensor 53.

-Coordinate transformation-
The coordinate conversion process 82 uses the position and angle data of each part of the vehicle body calculated in the attitude calculation process 81 to convert the output points of the measuring device 70 from values in the sensor coordinate system 300 to values in the vehicle body coordinate system 400. It is processing. The point group output by the measuring device 70 is a set of three-dimensional point data Ps (Xps, Yps, Zps) indicated by the sensor coordinate system 300. Point data (Xps, Yps, Zps) in the sensor coordinate system 300 is converted to point data Pv (Xpv, Ypv, Zpv) in the vehicle body coordinate system 400 using the following (Formula 1) to (Formula 3).

Here, Rsv is a rotation matrix that converts the coordinates of the sensor coordinate system 300 to the coordinates of the vehicle body coordinate system 400, and αs, βs, and γs in the rotation matrix Rsv are It is the angle of inclination. When the measuring device 70 is fixed to the hydraulic excavator 1, αs, βs, and γs are obtained by, for example, measuring the position and angle of the measuring device 70 in the vehicle body coordinate system 400, and stored in the ROM 71 or the storage device 57 in advance. It can be saved in .

θsw is the turning angle of the rotating body 7, and is calculated in the attitude calculation process 81.

Tsv is a translation vector that starts at the origin of the vehicle body coordinate system 400 and ends at the origin of the sensor coordinate system 300. The components Lsx, Lsy, and Lsz of Tsv are equal to the origin coordinates of the sensor coordinate system 300 in the vehicle body coordinate system 400. When the measuring device 70 is a fixed type, the origin position of the sensor coordinate system 300 in the vehicle body coordinate system 400 is immovable, so Tsv (Lsx, Lsy, Lsz) is measured in advance and stored in the ROM 71 or the storage device 57 in advance. You can save it.

- Transportation vehicle detection -
The transportation vehicle detection process 83 is a process of calculating Pdump, which is the position and angle of the tray T of the transportation vehicle 200, from the output point group of the measuring device 70 that has been converted to the value of the vehicle body coordinate system 400 in the coordinate conversion process 82. . In this transport vehicle detection process 83, it is also determined whether the point group is not acquired, the driving vehicle is not detected, the transport vehicle is detected, or the tray is detected. The flow of the transportation vehicle detection process 83 will be described later using FIGS. 1 and 13. In the transportation vehicle detection process 83, the position and angle of the transportation vehicle 200 are specifically acquired based on the following principle.

FIG. 9 is a side view showing the reference coordinate system of the transport vehicle, and FIG. 10 is a plan view. As shown in FIGS. 9 and 10, the X"Y"Z" right-handed orthogonal coordinate system whose origin is the point at the center of the vehicle width on the central axis of the rear wheel of the transport vehicle 200 is defined as the vehicle body coordinate system 600 of the transport vehicle 200. The X" axis is the direction from the origin toward the front of the vehicle, the Y" axis is the direction from the origin to the left along the wheel axis, the Z" axis is orthogonal to the X"Y" axis, and the positive direction is from the origin. The direction toward the top of the vehicle is defined as the positive direction. In this embodiment, the position and angle of the tray T of the transport vehicle 200 are specified based on four points Pd1, Pd2, Pd3, and Pd4 located at the four corners of the tray T of the transport vehicle 200 in plan view. In the transport vehicle detection process 83, points Pd1 to Pd4 are extracted from the output point group of the measuring device 70 converted to the vehicle body coordinate system 400 of the hydraulic excavator 1, and the coordinates of these points Pd1 to Pd4 are determined by the coordinates of the tray T of the transport vehicle 200. It is obtained as position and angle data (Pdump). The transportation vehicle detection process 83 is executed at a predetermined period (for example, about 100 ms to 1 sec).

- Situation Judgment -
The situation determination process 84 determines whether "running" or "turning" based on the lever operation signal acquired via the control device 40, the position and angle of each part calculated by the attitude calculation process 81, and the turning speed of the rotating body 7. This is a process to determine whether the condition falls under any of the conditions of ``medium''. The flow of the situation determination process 84 will be described later using FIG. 12.

-Detection result correction-
The detection result correction process 85 is a process for correcting the data of the detection result of the tray T acquired in the transport vehicle detection process 83, as necessary depending on the situation determined in the situation determination process 84. The final tray T detection result data obtained through the detection result correction process 85 is output to the control device 40 and the monitor 55, used as basic data for controlling the aircraft by the control device 40, and also displayed on the monitor 55. The operator will be notified. A specific example of the detection result correction process 85 will be described later.

7. Operation 7-1. Series of Operations FIG. 11 is a flowchart showing an example of a series of operations of the hydraulic excavator including tray detection processing. FIG. 11 illustrates the operational flow of the hydraulic excavator 1 during loading work.

(Step S101)
First, in the hydraulic excavator 1, it is confirmed whether the transport vehicle 200 is stopped at a position where the transported object can be loaded. This stopping of the transport vehicle 200 is performed by, for example, a processing device of the hydraulic excavator 1 by receiving a transport vehicle stop notification signal from a vehicle allocation management system of the transport vehicle 200 such as an FMS (Fleet Management System) used at a work site. 54 can be confirmed. In this case, the hydraulic excavator 1 is equipped with a communication device (not shown) for wireless communication with the server of the vehicle allocation management system. Alternatively, the operator may visually confirm that the transport vehicle 200 has stopped at a position where the transported object can be loaded, and manually input the fact that the transport vehicle 200 has stopped to the processing device 54. That is, based on the input signal input by the operator, the processing device 54 can confirm whether the transport vehicle 200 is stopped at a position where the cargo can be loaded.

(Step S102)
After confirming that the transport vehicle 200 has stopped at a position where the transported object can be loaded, the hydraulic excavator 1 detects the start of the loading operation. The start of the loading work can be detected by, for example, detecting the operation of the operating levers 22 and 23 by the processing device 54 based on the outputs of the sensors 52e and 52f. Note that the start of the loading work is not limited to this method, and can also be detected by receiving an excavation start command from a control such as FMS, for example. Furthermore, this data is based on data obtained through deep learning in advance of the movements of the hydraulic excavator 1 associated with the start of loading work, such as the swinging body 7 turning toward the transport vehicle 200 at the start of loading work. It is also possible to adopt a configuration in which the start of the loading work is detected based on the operation of the operating levers 22 and 23.

(Step S103)
When the start of the loading operation is detected, the hydraulic excavator 1 uses the processing device 54 to acquire current data on the position and angle of the tray T of the transport vehicle 200 into which the cargo is to be loaded, and outputs it to the control device 40 . The procedure for acquiring data on the position and angle of the tray T in step S103 will be described later.

(Step S104)
After acquiring the data on the position and angle of the tray T, the hydraulic excavator 1 uses the control device 40 to execute loading control according to the data on the tray T. In the loading control, the operation of at least one of the revolving body 7 and the front working machine 2 (boom 8 and arm 9) is controlled based on the data on the position and angle of the tray T acquired in step S103. As an example, when a swing operation signal is input, not only the swing but also the boom raising according to the swing speed is automatically or semi-automatically commanded even if the boom is not raised (or even if the operation amount is smaller than appropriate). Ru. With this loading control, the front working machine 2 (particularly the bucket 10) moves between the excavation position and the discharge position (earth discharge position) without colliding with the tray T.

(Step S105)
Separately from loading control, the hydraulic excavator 1 determines whether the loading operation for the transport vehicle 200 currently parked nearby is complete. For example, the operator visually confirms the loading status of the objects to be transported on the tray T, and when finishing the loading work on the tray T, performs a specific operation with the operating levers 22, 23 or the monitor 55, etc. to complete the loading work. The end of the process is input to the processing device 54. Based on the signal input in conjunction with this, the processing device 54 can determine the end of the loading operation. Note that the bucket loading amount of the front working machine 2 is measured, for example, by the pressure of the boom cylinder 11, etc., and when the cumulative loading amount to the transport vehicle 200 reaches a specified value, the processing device 54 determines the end of the loading work. It is also possible to do this. In addition, the processing device 54 receives measurement data from a loaded weight measuring device (for example, a load cell) mounted on the transportation vehicle 200 from the transportation vehicle 200, and determines the end of the loading operation when the measurement data reaches a specified value. It is also possible to do this. If the processing device 54 determines that the loading work for the stopped transport vehicle 200 has not been completed, the hydraulic excavator 1 returns the procedure from step S105 to step S102 and starts the loading work for the currently stopped transport vehicle 200. This series of processing is repeated.

(Step S106)
When the processing device 54 determines that the loading work for the stopped transport vehicle 200 has been completed, the hydraulic excavator 1 determines that the operation has ended. The determination of the end of the operation is made by the processing device 54, for example, based on a signal input (or input stopped) when the operator (in the case of an unmanned vehicle, the administrator) performs an operation to stop the prime mover 103 of the hydraulic excavator 1. Ru. When it is determined that the operation is continuing without an operation to end the operation, the hydraulic excavator 1 returns the procedure to step S101 and waits until the next transport vehicle 200 stops at a position where the transport object can be loaded. When the operation to end the operation is performed, the hydraulic excavator 1 executes the end processing (delay processing) of the processing device 54 and the control device 40, and ends the flow shown in FIG. 11.

7-2. Tray Position Acquisition The above-described procedure for acquiring data on the position and angle of the tray T (S103) will be explained. In the procedure for acquiring data on the position and angle of the tray T, the processing device 54 executes the flows shown in FIGS. 12 to 14 and 19. 12 and 13 are the flow of situation determination, FIG. 14 is the flow of tray detection processing, and FIG. 19 is the flow of detection result correction processing. Specifically, the flow in FIG. 12 represents the processing procedure for determining the situation (FIG. 8), and determines whether or not tray detection processing can be executed based on the displacement of the measuring device 70 (that is, whether "turning" or "traveling" in FIG. 5) is possible. ” ) is determined. The flow in FIG. 13 shows the procedure of the transportation vehicle detection process 83 (FIG. 8), and corresponds to the situations of "point cloud not acquired", "transportation vehicle not detected", "transportation vehicle detected", and "tray detected". The processing device 54 determines whether or not to do so. The tray detection process is executed in the process of determining whether the situation is "tray detection" or "transport vehicle detection", and the procedure will be separately explained with reference to FIG. 14. Further, the flow in FIG. 19 represents the procedure of the detection result correction process 85 (FIG. 8). These processes executed in the procedure for acquiring data on the position and angle of the tray T (S103) will be sequentially explained below.

7-2. Procedure of the situation determination process 84 (FIG. 8) FIG. 12 is a flowchart showing an example of the procedure of the situation determination process (FIG. 8).

(Step S201)
In the situation determination process 84, the processing device 54 first obtains operation signals input from the sensors 52a to 52f (FIG. 2) in response to the operation of the operation levers 22 and 23 by the operator.

(Step S202)
The processing device 54 determines whether the hydraulic excavator 1 is traveling based on the input operation signal. In this embodiment, it is determined whether or not the vehicle is running based on whether or not the running operation signals inputted from the sensors 52e and 52f are equal to or higher than a preset threshold value Vth.

(Step S203)
If it is determined in step S202 that the travel operation signal is equal to or higher than the threshold value Vth, the processing device 54 determines that the hydraulic excavator 1 is "travelling" and indicates that the current status is "travelling". The data shown in FIG. 12 is stored in the RAM 72, and the processing specified in the situation data table 800 is executed as appropriate, and the flow of the situation determination processing 84 in FIG. 12 is completed.

(Step S204)
On the other hand, if it is determined in step S202 that the traveling operation signal is less than the threshold value Vth, the processing device 54 determines that the hydraulic excavator 1 is not traveling (the traveling body 5 is stationary with respect to the ground). , the current turning speed calculated in the attitude calculation process 81 is read from the RAM 72.

(Step S205)
After acquiring the current turning speed in step S204, the processing device 54 determines whether the hydraulic excavator 1 (swinging structure 7) is turning based on the current turning speed. In this embodiment, it is determined whether or not the vehicle is turning based on whether the current turning speed is equal to or higher than a preset threshold value Ωth.

(Step S206)
If it is determined in step S205 that the turning speed is equal to or higher than the threshold Ωth, the processing device 54 determines that the hydraulic excavator 1 is “turning” on the rotating structure 7, and that the current situation is “turning”. Data indicating that there is something is stored in the RAM 72, and the process specified in the situation data table 800 is executed as appropriate, and the flow of the situation determination process 84 in FIG. 12 is ended.

(Step S207)
On the other hand, if it is determined in step S205 that the turning speed is less than the threshold value Ωth, the processing device 54 determines that the hydraulic excavator 1 is not turning (the rotating body 7 is stationary with respect to the traveling body 5). Then, the current detection result obtained by the transport vehicle detection process 83 (FIG. 13) is obtained.

(Step S208)
After acquiring the detection result by the transport vehicle detection process 83 (FIG. 13) in step S207, the processing device 54 updates the situation determination data stored in the RAM 72 with the acquired detection result, and end.

7-3. Procedure of Transport Vehicle Detection Process 83 (FIG. 8) FIG. 13 is a flowchart showing an example of the procedure of transport vehicle detection process (FIG. 8). This flow is executed by the processing device 54 separately (for example, in parallel) with the transport vehicle detection process 83 of FIG. 12. Alternatively, the detection result acquisition process (S207) may be executed when neither "running" nor "turning" is determined in the situation determination process 84 of FIG. 12. In this case, the processes of steps S304 and S305 in FIG. 13 (described later) are unnecessary.

(Step S301)
In the transport vehicle detection process 83, the processing device 54 first acquires the latest point cloud data measured by the measuring device 70. For example, the processing device 54 converts the point cloud data output by the measuring device 70 from the sensor coordinate system 300 to the vehicle body coordinate system 400 values in the coordinate conversion process 82 (FIG. 8), temporarily stores it in the RAM 72, and 11, the latest data is read at the start of the process in step S103.

(Step S302)
Next, the processing device 54 compares the output time of the latest point cloud data acquired in step S301 (the time output from the measurement device 70) with the current time, and calculates the elapsed time from the output time of the point cloud data to the current time. It is determined whether the time is less than or equal to a preset time ΔT (for example, 1 to 3 seconds). For example, a time stamp given by the processing device 54 when the point cloud data is input from the measuring device 70 can be used as the output time of the point cloud data. Furthermore, the real time time measured by the processing device 54 is used as the current time.

(Step S303)
If it is determined in step S302 that the elapsed time is greater than or equal to the set time ΔT, the processing device 54 determines that point cloud data has not been input from the measurement device 70 for a certain period of time (that is, greater than or equal to the set time ΔT). The situation is determined to be. Then, the processing device 54 stores data indicating that the current situation is "point cloud not acquired" in the RAM 72, and executes the processing specified in the situation data table 800 as appropriate to carry out the transport vehicle detection process shown in FIG. 83 ends.

(Step S304)
If it is determined in step S302 that the elapsed time is less than the set time ΔT, that is, if the point cloud data is being acquired in real time, the processing device 54 performs the determination in the situation determination process 84 (FIG. 12). Get current status data. Here, specifically, data indicating whether the vehicle is "running" or "turning" is read from the RAM 72 by the processing device 54.

(Step S305)
After reading the determination of the situation determination process 84 (FIG. 12), the processing device 54 determines whether or not the detection process of the transport vehicle 200 and the tray T can be executed based on the situation data table 800 (FIG. 5). If the current state is "running" or "turning", the detection process cannot be executed, and the processing device 54 ends the transport vehicle detection process 83 in FIG. 13. On the other hand, the determination in step S305 is executed on the condition that the situation is not "point cloud not acquired", so if the situation is neither "travelling" nor "turning", the processing device 54 shifts the procedure to detection processing. .

(Step S306)
If it is determined in step S305 that the hydraulic excavator 1 is neither "travelling" nor "turning" and the procedure moves to detection processing, the processing device 54 detects objects around the hydraulic excavator 1 from the point cloud data acquired in step S301. To detect. For example, the processing device 54 calculates the distance of each point in the point cloud data, extracts a set (cluster) of points whose distance to adjacent points is less than or equal to a predetermined distance, and surrounds clusters with more points than the predetermined value. Detected as an object. At this time, points estimated to be the measured points on the ground are removed from the point cloud data acquired in step S301, and the remaining data is used to detect surrounding objects. For example, it is possible to extract a plane corresponding to the ground from the point cloud data, and remove the points included in the extracted plane by estimating that they are the points from which the ground was measured. Then, after removing the measured points on the ground, for each point that remains, calculate the distance between adjacent points (distance between two points), and select a set of points whose distance between two points is smaller than a predetermined value. Extract the object as a point cloud cluster. For example, the Euclidean Cluster Extraction algorithm can be applied to extract point cloud clusters to be detected as objects. Further, for example, the RANSAC (RANdom SAmple Consensus) algorithm can be applied to extract the ground.

(Step S307)
After completing the surrounding object detection process, the processing device 54 determines whether or not the transport vehicle 200 is included in the detected surrounding objects. For example, the smallest cube surrounding the cluster of points detected in the process of step S306 is calculated, and if one side of the cube is larger than a preset size (for example, 5 m), It can be determined that the point cloud cluster is data for detecting the transport vehicle 200. In addition, it is also desired to prevent the point cloud glaster that measures dust from being mistakenly detected as the transport vehicle 200. In that case, for example, if one side of the cube surrounding the point cloud cluster is larger than the maximum expected size of the transport vehicle or a value estimated larger than that (for example, 10 m), the point cloud cluster will be larger than the set size. Processing device 54 may be programmed so that transport vehicle 200 is not detected even if

As described above, the determination of the transport vehicle 200 is not limited to the size of the point cloud cluster, but is also determined based on whether a characteristic geometric shape of the transport vehicle 200 (for example, the circular shape of a tire) is extracted. You can also do that.

(Step S308)
If it is determined in step S307 that the transport vehicle 200 is not detected, the processing device 54 determines the situation as "transport vehicle not detected", and indicates that the current situation is "transport vehicle not detected". The data is stored in the RAM 72, the processing specified in the situation data table 800 is executed as appropriate, and the transportation vehicle detection processing 83 of FIG. 13 is completed.

(Step S309)
If it is determined in step S307 that the transport vehicle 200 has been detected, the processing device 54 executes tray detection processing. In this tray detection process, Pdump (Pd1 to Pd4) described above with reference to FIGS. 9 and 10 is calculated for the point cloud cluster estimated to be the transport vehicle 200. The tray detection process (step S309) will be explained later using FIG. 14.

(Step S310)
After executing the tray detection process, the processing device 54 determines whether the tray detection result is successful or not. The validity of the tray detection result is determined by calculating the length in the width direction (that is, the distance between the two left and right sides) of the Pdump obtained in the tray detection process, and checking whether it is larger than a predetermined size. It can be determined by The predetermined size at this time is, for example, equivalent to the width of the transport vehicle 200. Further, if there is a possibility that the bucket of the hydraulic excavator 1 enters the measurement range of the measuring device 70, the side surface of the bucket may be mistakenly detected as the side surface of the transport vehicle, so it is desirable to set the width to at least the bucket width.

(Step S311)
If it is determined in step S310 that the tray T has been detected, the processing device 54 determines the situation as "tray detected" and sends data indicating that the current situation is "tray detected" and the transport vehicle 200 or The data of the detection result of the tray T is stored in the RAM 72, and the processing specified in the situation data table 800 is executed as appropriate, and the transport vehicle detection processing 83 of FIG. 13 is ended.

(Step S312)
If it is determined in step S310 that the tray T is not detected, the processing device 54 determines the situation as "transport vehicle detected" and stores data indicating that the current situation is "transport vehicle detected" in the RAM 72. Then, the process specified in the situation data table 800 is executed as appropriate, and the transport vehicle detection process 83 in FIG. 13 is ended.

7-4. Tray Detection Processing FIG. 14 is a flowchart showing an example of the procedure of tray detection processing. In the tray detection process, the left and right sides of the tray T of the transport vehicle 200 (for example, the left and right inner wall surfaces of the tray T) are extracted, and Pdump is output as the feature point of the tray T, as described below.

(Step S401)
When starting the tray detection process, the processing device 54 first acquires data of the point cloud cluster extracted as the transport vehicle 200.

(Step S402)
After acquiring the data of the point group cluster extracted as the transport vehicle 200, the processing device 54 calculates the normal vector of each point of the point group cluster. The normal vector of each point can be determined using a general method. For example, by applying a plane to a local point group using the least squares method, a vector orthogonal to the plane can be determined as the normal vector.

(Step S403)
After calculating the normal vector of each point, the processing device 54 determines whether the distance to the adjacent point is less than or equal to a predetermined value and the normal vector is equal to that of the adjacent point (or the difference between the normal vectors is less than or equal to the predetermined value). Extract point cloud clusters (grouping processing). For example, a Region Growing algorithm can be applied to the grouping process using normal lines.

(Step S404)
After performing the grouping process, the processing device 54 calculates an approximate plane for each grouped point cloud cluster, and calculates a normal vector of the approximate plane of each point cloud cluster. For example, the RANSAC algorithm can be applied to calculate the approximate plane.

(Step S405)
After calculating the approximate plane, the processing device 54 removes unnecessary point group clusters, specifically, point group clusters whose normal vector calculated in step S404 is parallel to the gravity direction vector. For example, the inner product of the normal vector and the Z-axis direction vector of the vehicle body coordinate system 400 is calculated, and the normal vector is determined to be parallel to the gravity direction vector based on whether the calculated inner product is greater than or equal to a preset threshold. It is determined whether or not there is. If the pitch angle θpitch (FIG. 9), roll angle θroll (FIG. 9), and yaw angle θyaw (FIG. 10) of the transport vehicle 200 can be obtained, the inner product may be calculated by taking these pitch angles and roll angles into account.

(Step S406)
After removing unnecessary point cloud clusters, the processing device 54 selects points that form a pair (for example, the approximate planes are parallel) as the left and right inner wall surfaces of the tray T from the point cloud clusters forming the approximate plane calculated in step S404. Determine the presence or absence of group clusters. For example, for each point cloud cluster extracted in step S404, the combination in which the absolute value of the inner product of the normal vectors of each approximate plane is greater than or equal to a preset threshold and the inner product is the largest is the combination of the two left and right surfaces of the tray T. It is calculated as

Note that the tray of the transport vehicle 200 has various shapes. 15 to 18 are schematic cross-sectional views of the tray T of the transport vehicle taken along a vertical plane extending in the vehicle width direction. The tray T shown in Fig. 16 is an example in which the left and right sides are flat and parallel; however, some trays T have flat left and right sides as shown in Fig. 15, but the distance between the surfaces increases toward the top, and the left and right sides are flat. The sides of the plane may not be parallel. Furthermore, as shown in FIGS. 17 and 18, some or all of the left and right side surfaces of the tray T may be formed with curved surfaces. In this case, instead of comparing the normals of the approximate planes calculated from the point cloud cluster, for example, the projection vectors of the normal vectors of the approximate planes to the XY plane in the vehicle body coordinate system 400 are compared, and the left and right sides of the tray T are Pairs constituting two sides of can be extracted.

(Step S407)
If it is determined in step S406 that two planes forming a pair have not been extracted (for example, if there is no pair of approximate planes for which the absolute value of the inner product is greater than or equal to the threshold), the processing device 54 proceeds to step S402 (FIG. 13). Shift the procedure (finish tray detection in FIG. 14).

(Step S408)
If it is determined in step S406 that two sides forming a pair have been extracted, the processing device 54 calculates the vertices of the four corners of the tray T (points Pd1 to Pd4 in FIG. 10). For example, extract two point groups (point groups lined up in a line) that form the upper end of each point group cluster, and set both ends (start point and end point) of these two point groups as points Pd1 to Pd4. Extract. At this time, since the transport vehicle 200 has already been detected, the processing device 54 can detect the front and rear of the tray T.

(Step S409)
After calculating the four corner points Pd1 to Pd4 of the tray T, the processing device 54 moves the procedure to step S401 (FIG. 13) (ends the tray detection in FIG. 14).

7-5. Detection Result Correction Processing Procedure FIG. 19 is a flowchart showing an example of the detection result correction processing procedure (FIG. 8). The detection result of the tray T by the transport vehicle detection process 83 is corrected in a predetermined case as described below.

(Step S501)
After the transport vehicle detection process 83 and the situation determination process 84 are completed, the processing device 54 reads the data of the situation determination and tray detection results obtained in the transport vehicle detection process 83 and the situation determination process 84 from the RAM 72.

(Step S502)
Next, the processing device 54 determines whether the detection result can be output based on the situation data table 800. In other words, it is determined whether the current situation is "transport vehicle detection", "tray detection", "turning", or any other situation.

(Step S503)
When the current situation is "transport vehicle detection", "tray detection", or "turning", the processing device 54 corrects the coordinates of the Pdump points Pd1 to Pd4 detected in the transport vehicle detection process 83.

20 and 21 are explanatory diagrams of an example of correction of tray detection data. The tray T is generally rectangular when viewed from above. However, for example, if the four corners of the tray T cannot be measured by the measuring device 70 due to interference from transported objects such as earth and sand loaded on the tray T, or if the bias of the point cloud output from the measuring device 70 has an effect, etc. The tray T thus formed may become trapezoidal as shown in FIGS. 20 and 21. Therefore, on the premise that the actual tray T is rectangular, the points Pd1 to Pd4 of the trapezoidally detected tray T are corrected so that they become the vertices of the rectangle. In FIGS. 20 and 21, the vertices of the tray T before correction in step S503 are referred to as points Pd1, Pd2, Pd3, and Pd4, and the vertices of the tray T after correction are referred to as points Pd1', Pd2', Pd3', and Pd4'. indicate.

The corrections at the points Pd1 to Pd4 are executed by the processing device 54 using, for example, the following (Equations 4) to (Equations 8). In the examples of FIGS. 20 and 21, the line segment Pd3-Pd4 and the line segment Pd1-Pd2 are parallel, but have different lengths. In the same example, the length of the shorter line segment Pd1-Pd2 is adjusted to the length of the longer line segment Pd3-Pd4, and the points Pd1 and Pd2 are changed to the point Pd1' so that the points Pd1' to Pd4' form a rectangle. , Pd2' is illustrated. In FIG. 20, point Pd1 is corrected to point Pd1', and further in FIG. 21, point Pd2 is corrected to point Pd2'.

(Step S504)
After correcting the vertices to a rectangular shape, the processing device 54 calculates the detected tray length Lrecg as the longitudinal length of the tray T that is virtualized by the corrected points Pd1' to Pd4'. Lrecg is equal to the length of vector A (FIG. 20) and vector D (FIG. 21) in Pdump after correction.

(Step S505)
After calculating the detected tray length Lrecg, the processing device 54 determines whether Lrecg is less than a preset threshold Lth.

(Step S506)
If Ldump is less than Lth, the processing device 54 corrects the coordinates of points Pd1 to Pd4 so that Ldump becomes a preset tray length Ldump.

FIG. 22 is an explanatory diagram of an example of tray length correction. As shown in the figure, the detection tray length Lrecg is extended in the traveling direction (forward) of the transport vehicle 200. Specifically, the coordinates of points Pd2' and Pd3' are corrected to points Pd2'' and Pd3'' using (Equation 9).

(Step S507)
After correcting the tray length, or if Ldump≧Lth in step S505, the processing device 54 outputs the corrected position and angle data (Pdump) of the tray T to the control device 40 and the monitor 55. , the procedure of FIG. 19 regarding the detection result correction process 85 is completed.

8. Display Example FIG. 23 is a diagram showing a display example of the detection result of the tray T. In the example shown in the figure, the line segment Pd1-Pd2 and the line segment Pd3-Pd4 are added to the point cloud measured by the measuring device 70 as data (Pdump) of the position and angle of the transport vehicle 200 detected by the processing device 54. are displayed overlapping each other. This allows the operator to be informed that the upper end surface of the side surface of the tray T has been detected from the point cloud. The operator riding the hydraulic excavator 1 can visually check the detection state of the tray T by the processing device 54 from this display, and can use this as a reference for, for example, the operation timing of loading control.

Preferably, among the point groups measured by the measuring device 70, the point groups extracted as the left and right sides of the tray T in the transport vehicle detection process 83 are changed in color or density as shown in the figure. It is displayed to distinguish it from other parts. You can visually check whether the side plane of the tray is being detected properly.

Further, in the example shown in the figure, a frame F whose display color is changed according to the situation defined in the situation data table 800 is displayed. For example, by displaying the frame F in blue or green while the processing device 54 is detecting the tray T, the operator can intuitively confirm that the tray T has been detected. Furthermore, if the transportation vehicle 200 is not detected due to dust or the like, for example, by displaying the frame F in red, the operator can intuitively confirm that the transportation vehicle 200 has not been detected. By notifying the operator of the tray detection status in this manner, the operator can appropriately determine whether or not automatic control can be applied to the operation of the hydraulic excavator 1. Further, as shown in the figure, the situation can be specifically notified using text information I such as "Tray is being detected."

In addition, when displaying the detection status by the processing device 54 according to the status data table 800, while the output of the current position of the tray T is stopped (for example, "point cloud not acquired", "transport vehicle not detected", It is also possible to display on the monitor 55 that the current position of the tray T has not been detected.

9. Effects (1) Extract two horizontally opposing surfaces from the point cloud output by the measurement device 70, determine whether the distance between the two extracted surfaces is longer than a preset distance, and determine whether the distance between the two surfaces is longer than the preset distance. If it is longer, a tray T with two sides as left and right walls is detected. By detecting the tray T based on the characteristics common to the transport vehicle 200 in this way, the tray T can be detected without relying on machine learning. Even if the transport vehicle has no data, the position of the tray T can be detected with high accuracy. Of course, the tray T of the transport vehicle 200 can also be detected at a new work site. Since new machine learning is not required, the time required to adjust the processing device 54 at the work site can also be reduced. By applying the detection data of the tray T to the control of the loading operation by the hydraulic excavator 1, the productivity of the earth and sand transport work is also improved.

(2) By setting the set distance to be compared with the distance between the two extracted surfaces to be at least larger than the width of the bucket 10 of the front working machine 2, the bucket 10 is measured by the measuring device 70 and falsely detected as a surrounding object. It is possible to prevent this from happening.

(3) Furthermore, in this embodiment, an object larger than a preset size or an object detected including a circle corresponding to a running wheel is used as the transport vehicle 200 from the point cloud output by the measuring device 70. Then, from the point group of this transportation vehicle 200, two surfaces facing each other in the horizontal direction are extracted. By detecting the tray T on the premise that the transport vehicle 200 is detected in this manner, the tray T, which is a component of the transport vehicle 200, can be detected rationally.

(4) If the point cloud is not input from the measurement device 70, if the transport vehicle 200 is not detected, or if the effectiveness of the tray detection process is not ensured, the tray detection process can be interrupted to reduce the load on the processing device 54. can be reduced.

(5) Although the tray detection process itself can be performed while the vehicle is turning or traveling, the measuring device 70 is displaced during the revolution or traveling, so the accuracy of calculating the coordinates of the point group during that time is reduced. Therefore, by interrupting the tray detection process when the hydraulic excavator 1 is turning or running as in the present embodiment, it is possible to suppress the output of detection results that lack accuracy and to suppress the calculation load on the processing device 54. be able to.

(6) Since an algorithm is specified for each situation specified in the situation data table 800, even if the tray T is no longer detected due to the loading of objects, for example, while the transport vehicle 200 is detected. , the latest detection data is output to the control device 40 and monitor 55. In this way, by predetermining whether or not tray detection processing can be executed and how to manage the detection results, it is possible to accurately detect the tray T, reduce the load on the processing device 54, and take flexible measures depending on the situation. be able to. In addition, even though the tray T is not detected during "transport vehicle detection" or "while turning", valid tray detection data can be retained, increasing the chances of valid tray detection data being output, increasing productivity. will improve.

(7) If the lengths in the longitudinal direction of the two surfaces extracted during the tray detection process are different, the detection data of the shorter surface is corrected in accordance with the longer surface. For example, even if there is a difference in the proportion of the area detected by the measuring device 70 between the left and right sides of the tray T due to interference from a transported object, since the left and right sides of the tray T are actually symmetrical, the tray T The detection area can be rationally expanded.

(8) Furthermore, the tray T of the transport vehicle 200 at a mining site often has a bottom that slopes downward toward the front, as shown in the schematic diagram of FIG. 9, so that the loaded earth and sand moves toward the front. In this case, as the loading of earth and sand into the tray T progresses, the front portion of the left and right inner wall surfaces of the tray T cannot be measured by the measuring device 70, and in step S504, Lrecg becomes much larger than the actual length of the tray T. It may be calculated shorter.

Therefore, in this embodiment, when the length of the detected tray T, that is, the detected tray length Lrecg is shorter than the threshold value Lth, the detected tray length Lrecg is corrected to the set tray length Ldump. In this case, by correcting the detection tray length Lrecg by extending it forward to reflect the fact that the front part of the tray T is more difficult to detect as described above, it is possible to rationally detect the entire tray T. can.

(9) Since the measuring device 70 is installed on the revolving body 7, especially in the case of this embodiment, above the driver's cab 20, the transport vehicle 200 can be reasonably measured with the measuring device 70.

(10) By displaying the detection data of the tray T on the monitor 55 overlapping the point cloud from the measuring device 70, it is possible to notify the operator that the upper end surface of the side surface of the tray T has been detected from the point cloud. The operator riding the hydraulic excavator 1 can visually check the detection state of the tray T by the processing device 54 from this display, and can use this as a reference for, for example, the operation timing of loading control.

(11) Among the point groups measured by the measuring device 70, the point groups extracted as the left and right sides of the tray T in the transportation vehicle detection process 83 are highlighted and displayed with color, density, etc. Thereby, the operator can visually confirm whether the processing device 54 is appropriately detecting the tray T from the shape of the highlighted point group, etc. Displaying a frame F whose display color changes depending on the situation on the monitor 55 and text information I specifically explaining the situation can also effectively notify the operator of the situation.

(Second embodiment)
FIG. 24 is a functional block related to tray detection processing by a processing device provided in a working machine according to the second embodiment of the present invention. In the figure, elements that are the same as or correspond to those in the first embodiment are given the same reference numerals as those in the previous drawings, and a description thereof will be omitted.

The working machine according to this embodiment is equipped with a communication device 60 that receives position data of the transport vehicle 200. The processing device 54 receives position data of the transport vehicles 200 around the hydraulic excavator 1 from a vehicle allocation management system (for example, FMS) via the communication device 60 . The processing device 54 of this embodiment executes the tray detection process only when it is determined that the transportation vehicle 200 is located in a preset area based on the position data of the transportation vehicle 200 received by the communication device 60. . In the first embodiment, the tray detection process (the tray position acquisition procedure in step S103 in FIG. 11) is periodically executed regardless of the presence or absence of the transport vehicle 200. In contrast, the processing device 54 of the present embodiment performs tray detection processing (step S103 in FIG. Execute the tray position acquisition procedure). In other respects, this embodiment is similar to the first embodiment.

According to the present embodiment, in addition to the same effects as the first embodiment, by limiting the opportunity for tray detection processing to be executed only when the transport vehicle 200 is located around the hydraulic excavator 1, Compared to the first embodiment, the processing load on the processing device 54 is reduced. In addition, since the tray detection process is executed in a situation where the transport vehicle 200 is actually located near the hydraulic excavator 1 based on the dispatch data received via the communication device 60, the tray T detection accuracy is also improved.

(Modified example)
Note that the embodiments of the present invention are not limited to the above, and include various modifications and combinations without departing from the gist of the invention. For example, except for the configuration (including the algorithm) for obtaining the essential effect (1) described above, other configurations may be omitted or changed as appropriate. To give a few examples, for example, the hydraulic excavator 1 is configured to be operated by an operator in the cab 20, but as long as effect (1) is obtained, the hydraulic excavator 1 can be a manned aircraft or an unmanned aircraft. I don't mind. In the case of an unmanned aircraft, the design is changed so that the processing device 54 receives, for example, an operation signal received via wireless communication from a control station, etc., rather than an operation signal accompanying an operation by an operator on board. Further, as the measuring device 70, a type that scans an object by swinging the optical axis can also be applied. In this case, as the rotation matrix Rsv for coordinate transformation from the sensor coordinate system 300 to the vehicle body coordinate system 400, a matrix that dynamically changes according to the movement of the optical axis is used. In addition, the threshold value Lth that is compared with the detected tray length Lrecg in the tray length correction and the set tray length Ldump that is the correction target do not have to be fixed values, and for example, the log data of the tray detection results of past tray detection processing may be used. It may be a value set based on.

DESCRIPTION OF SYMBOLS 1... Hydraulic excavator (work machine), 2... Front working machine, 5... Traveling body, 7... Swinging body, 10... Bucket, 19... Turning angle sensor, 20... Operator's cab, 52e, 52f... Sensor (traveling sensor), 54...Processing device, 55...Monitor, 60...Communication device, 70...Measuring device, 200...Transportation vehicle, Ldump...Setting tray length, Lth...Threshold value, T...Tray

Claims (13)

In a working machine that loads materials into the tray of a transportation vehicle,
A running body,
a revolving body rotatably provided on the upper part of the traveling body;
a front working machine attached to the revolving body;
a measuring device that measures objects around the work machine and outputs a point cloud indicating the measurement results;
a processing device that executes a detection process of the tray from the point cloud output by the measurement device;
The processing device includes:
extracting two horizontally opposing surfaces from the point cloud output by the measuring device;
Determine whether the distance between the two extracted surfaces is longer than a predetermined distance,
If it is determined that the distance between the surfaces is longer than the set distance, the tray with the two surfaces as left and right walls is detected, and when the tray is detected, the position of the tray is output. working machine. The working machine according to claim 1,
The working machine is characterized in that the set distance is set to be larger than the width of the bucket of the front working machine. The working machine according to claim 1,
The processing device includes:
Detecting an object larger than a preset size or an object containing a circle corresponding to a running wheel from the point cloud output by the measuring device as the transport vehicle,
A working machine characterized in that two horizontally opposing surfaces are extracted from the point group of the transport vehicle. The working machine according to claim 1,
The working machine is characterized in that the processing device interrupts the tray detection process when a point cloud is not input from the measurement device and when the transport vehicle is not detected. The working machine according to claim 1,
comprising a turning angle sensor that detects the angle of the rotating body with respect to the traveling body,
The measuring device is installed on the rotating body,
The working machine is characterized in that the processing device interrupts the tray detection process when it is determined that the rotating body is rotating based on the output of the rotating angle sensor. The working machine according to claim 1,
comprising a travel sensor that detects the travel of the traveling body,
The working machine is characterized in that the processing device interrupts the tray detection processing when it is determined that the working machine is running based on the output of the running sensor. The working machine according to claim 1,
The processing device includes:
retaining the latest detection data of the tray obtained in the tray detection process;
A working machine characterized in that even if the tray is no longer detected, the latest detection data is output while the transport vehicle is being detected. The working machine according to claim 1,
The processing device includes:
Compare the lengths of the two extracted surfaces in the longitudinal direction,
A working machine characterized in that when the two surfaces have different lengths in the longitudinal direction, the detection data of the shorter surface is corrected in accordance with the detection data of the longer surface. The working machine according to claim 1,
The processing device includes:
storing a threshold value set in advance as the length of the tray in the longitudinal direction;
Comparing the detected longitudinal length of the tray with the threshold,
A work machine characterized in that when the detected length of the tray is shorter than the threshold value, the length of the tray is corrected to a predetermined set tray length. The working machine according to claim 9,
The working machine is characterized in that the processing device corrects the detected length of the tray by extending it toward the front of the transport vehicle. The working machine according to claim 1,
A working machine, wherein the measuring device is installed on the revolving body. The working machine according to claim 1,
Equipped with a monitor located in the driver's cab,
The working machine is characterized in that the processing device causes the detection data of the tray to be displayed on the monitor in a superimposed manner with a point group from the measuring device. The working machine according to claim 1,
comprising a communication device that receives position data of the transport vehicle,
The processing device executes the tray detection process only when it is determined that the transportation vehicle is located in a preset area based on the location data of the transportation vehicle received by the communication device. and working machines.

Images