JP2024028438A - Work machine image display system and work machine image display method - Google Patents

Abstract

An object of the present invention is to suppress a decrease in work efficiency when working using a work machine equipped with a work tool. An image display system for a work machine includes an imaging device attached to a work machine having a work tool, an attitude detection device for detecting the attitude of the work machine, and a work target of the work machine. a distance detection device that obtains information on the distance of the object, information on the position of the work implement obtained using the attitude of the work implement, and information on the distance obtained from the distance information obtained by the distance detection device; using the position information to generate an image of a portion of the work object facing the work tool that corresponds to the work tool, synthesize it with the image of the work object captured by the imaging device, and display the image. and a processing device for displaying information on the device. [Selection diagram] Figure 1

Description

The present invention relates to an image display system for a working machine, a remote control system for a working machine, a working machine, and an image display method for a working machine.

As described in Patent Document 1, a technique for remotely controlling a working machine such as a hydraulic excavator is known.

Japanese Patent Application Publication No. 2004-294067

When remotely controlling a working machine, the displayed image is two-dimensional and lacks a sense of perspective when operating using an image from the operator's viewpoint of the working machine. Therefore, it becomes difficult to grasp the distance between the work target and the work machine, which may reduce work efficiency. Additionally, when an operator on board a work machine operates the work machine, depending on the skill level of the operator, it may be difficult to grasp the distance between the work machine and the work target, which may reduce work efficiency. .

An object of the present invention is to suppress a decrease in work efficiency when working using a work machine equipped with a work tool.

The present invention provides an imaging device attached to a working machine having a working tool, an attitude detection device for detecting the attitude of the working machine, and a distance detection device for obtaining information on the distance of the working machine to a work target. using a device, information on the position of the work implement obtained using the attitude of the work implement, and information on the position of the work target obtained from the information on the distance determined by the distance detection device, a processing device that generates an image of a portion of the work object facing the work tool that corresponds to the work tool, combines the image with an image of the work object captured by the imaging device, and displays the image on a display device; This is an image display system for working machines, including: It is preferable that the processing device generates an image of a portion corresponding to the working tool using the imaging device as a reference.

Preferably, the line image is a lattice including a plurality of first line images and a plurality of second line images intersecting the plurality of first line images.

It is preferable that the processing device determines an area occupied by the work implement in the image of the work object using the attitude of the work implement, and removes the obtained area from information on the shape of the work object.

The work tool is a bucket, and the processing device generates a line on the work object corresponding to the cutting edge of the bucket as an image of a portion of the work object facing the work tool corresponding to the work tool. Preferably, an image is generated.

The processing device generates an image of a straight line connecting one end side in the width direction of the blade edge of the bucket and the work object, and a straight line connecting the other end side in the width direction of the blade edge of the bucket and the work object. It is preferable to generate an image, combine it with an image of the work target captured by the imaging device, and display the synthesized image on a display device.

It is preferable that the processing device obtains spatial position information regarding the work tool or the work object and displays it on the display device.

The processing device is configured to determine the position of the work tool, the attitude of the work tool, the position of the work object, the relative attitude of the work object, the relative distance between the work tool and the work object, and the distance between the work tool and the work object. Preferably, at least one posture relative to the work object is determined and displayed on the display device.

Preferably, the processing device generates a line image along the surface of the work object using information on the position of the work object, combines the line image with the image of the work object, and displays the line image on the display device.

It is preferable that the imaging device, the attitude detection device, and the distance detection device be provided in the working machine, and that the processing device and the display device be provided in a facility provided with an operating device for remotely controlling the working machine.

In remotely controlling a working machine equipped with a working tool, the present invention provides a display device, information on the position of the working tool obtained using the attitude of the working tool, and a display device that the working machine includes. Corresponding to the work tool on the work object facing the work tool, using information on the position of the work object obtained from information on the distance to the work object of the work implement determined by the distance detection device. an image display system for a working machine, comprising: a processing device that generates an image of a portion using the imaging device as a reference, synthesizes it with an image of the work target captured by the imaging device, and displays the image on the display device; It is.

The present invention is a remote control system for a working machine, including the above-described image display system for a working machine and an operating device for operating the working machine included in the working machine.

The present invention is a working machine equipped with the above-described image display system for a working vehicle.

INDUSTRIAL APPLICATION This invention can suppress the decline in work efficiency when working using a working machine equipped with a working tool.

FIG. 1 is a diagram showing an image display system for a working machine and a remote control system for a working machine according to an embodiment. FIG. 2 is a perspective view showing a hydraulic excavator which is a working machine according to an embodiment. FIG. 3 is a diagram showing a control system of a hydraulic excavator, which is a working machine according to an embodiment. FIG. 4 is a diagram for explaining the coordinate system in the image display system and remote control system according to the embodiment. FIG. 5 is a rear view of the hydraulic excavator. FIG. 6 is a diagram illustrating the coordinate systems of the imaging device and the distance detection device. FIG. 7 is a flowchart of an example of control executed by the image display system and the remote control system. FIG. 8 is a diagram showing an imaging device, a distance detection device, and a work target. FIG. 9 is a diagram illustrating the occupied area. FIG. 10 is a diagram showing information on the shape of the work target with the occupied area removed. FIG. 11 is a diagram for explaining an image showing the position of a bucket on a work object. FIG. 12 is a diagram for explaining an image showing the position of the bucket on the work object. FIG. 13 is a diagram for explaining an image showing the position of the bucket on the work object. FIG. 14 is a diagram showing a grid image that is a reference image. FIG. 15 is a diagram showing a grid image that is a reference image. FIG. 16 is a diagram showing a work image. FIG. 17 is a diagram for explaining a blade edge position image when a loading shovel type work machine is used. FIG. 18 is a diagram illustrating a first modified example of the process for obtaining a blade edge position image. FIG. 19 is a diagram for explaining a second modified example of the process for obtaining a blade edge position image. FIG. 20 is a diagram for explaining a second modified example of the process for obtaining a blade edge position image. FIG. 21 is a diagram showing a control system of a hydraulic excavator according to a modification.

DESCRIPTION OF THE PREFERRED EMBODIMENTS Modes (embodiments) for carrying out the present invention will be described in detail with reference to the drawings.

<Overview of the image display system for working machines and the remote control system for working machines>
FIG. 1 is a diagram showing an image display system 100 for a working machine and a remote control system 101 for a working machine according to an embodiment. The image display system 100 for a working machine (hereinafter referred to as the image display system 100 as appropriate) is used to display the work target of the hydraulic excavator 1, more specifically, when an operator remotely controls the hydraulic excavator 1, which is a working machine. The imaging device 19 captures an image of the topographical surface that is the object of work by the work machine 2 of the hydraulic excavator 1, that is, the work object WA and the bucket 8 that is the work tool, and displays the obtained image on the display device 52. At this time, the image display system 100 displays a work image including an image 68 of the work target WA captured by the imaging device 19, a grid image 65, and an image 60 for indicating the position of the bucket 8 on the work target WA. The image 69 of is displayed on the display device 52.

The image display system 100 includes an imaging device 19, a posture detection device 32, a distance detection device 20, and a processing device 51. A remote control system 101 for a work machine (hereinafter referred to as the remote control system 101 as appropriate) includes an imaging device 19, an attitude detection device 32, a distance detection device 20, a work machine control device 27, a display device 52, and a processing unit. It includes a device 51 and an operating device 53. In the embodiment, the imaging device 19, posture detection device 32, and distance detection device 20 of the image display system 100 are provided in the hydraulic excavator 1, and the processing device 51 is provided in the facility 50. The facility 50 is a facility that remotely controls the hydraulic excavator 1 and manages the hydraulic excavator 1. In the embodiment, the imaging device 19, posture detection device 32, distance detection device 20, and work machine control device 27 of the remote control system 101 are provided in the hydraulic excavator 1, and the display device 52, processing device 51, and operation device 53 are provided in the facility 50. established in

The processing device 51 of the image display system 100 includes a processing section 51P, a storage section 51M, and an input/output section 51IO. The processing unit 51P is, for example, a processor such as a CPU (Central Processing Unit). The storage unit 51M is, for example, a RAM (Random Access Memory), a ROM (Read Only Memory), a hard disk drive, a storage device, or a combination thereof. The input/output unit 51IO is an interface circuit for connecting the processing device 51 and external equipment. In the embodiment, a display device 52, an operating device 53, and a communication device 54 are connected to the input/output unit 51IO as external devices. External devices connected to the input/output unit 51IO are not limited to these.

The processing device 51 processes information on the position of the bucket 8, which is a work implement, obtained using the attitude of the work equipment 2, and information on the position of the work object WA obtained from the distance information obtained by the distance detection device 20. An image of a portion of the work target WA facing the bucket 8 corresponding to the bucket 8 is generated using the imaging device 19 as a reference. Then, the processing device 51 combines the image with the image of the work target WA captured by the imaging device 19 and causes the display device 52 to display the combined image. The work target WA is a surface on which the work machine 2 of the hydraulic excavator 1 performs work such as excavation or earth leveling.

The display device 52 is exemplified by a liquid crystal display or a projector, but is not limited to these. The communication device 54 includes an antenna 54A. The communication device 54 communicates with the communication device 25 provided in the hydraulic excavator 1 to acquire information about the hydraulic excavator 1 and to transmit information to the hydraulic excavator 1.

The operating device 53 includes a left operating lever 53L installed on the left side of the operator, and a right operating lever 53R installed on the right side of the operator. The left operating lever 53L and the right operating lever 53R correspond to two-axis movements in the front, rear, left and right directions. For example, the operation of the right operation lever 53R in the front-rear direction corresponds to the operation of the boom 6 of the working machine 2 included in the hydraulic excavator 1. The left-right direction operation of the right operation lever 53R corresponds to the operation of the bucket 8 of the working machine 2. The operation of the left operation lever 53L in the front-back direction corresponds to the operation of the arm 7 of the working machine 2. The left-right operation of the left operating lever 53L corresponds to turning the upper rotating body 3 of the hydraulic excavator 1.

The operating amounts of the left operating lever 53L and the right operating lever 53R are detected by, for example, a potentiometer, a Hall IC, etc., and the processing device 51 generates a control signal for controlling the electromagnetic control valve based on these detected values. . This signal is sent to the work machine control device 27 via the communication device 54 of the facility 50 and the communication device 25 of the hydraulic excavator 1. The work machine control device 27 controls the work machine 2 by controlling the electromagnetic control valve based on the control signal. The electromagnetic control valve will be described later.

The processing device 51 acquires an input to at least one of the left operating lever 53L and the right operating lever 53R, and generates a command for operating at least one of the working machine 2 and the upper rotating body 3. The processing device 51 transmits the generated command to the communication device 25 of the hydraulic excavator 1 via the communication device 54. The work implement control device 27 included in the hydraulic excavator 1 acquires a command from the processing device 51 via the communication device 25, and operates at least one of the work implement 2 and the upper revolving structure 3 in accordance with the command.

The hydraulic excavator 1 includes a communication device 25, a work implement control device 27, a posture detection device 32, an imaging device 19, a distance detection device 20, antennas 21 and 22, and a global position calculation device 23. The work machine control device 27 controls the work machine 2. The communication device 25 is connected to the antenna 24 and communicates with a communication device 54 provided in the facility 50. The work machine control device 27 controls the work machine 2 and the upper revolving structure 3. The attitude detection device 32 detects the attitude of at least one of the working machine 2 and the hydraulic excavator 1. The imaging device 19 is attached to the hydraulic excavator 1 and images the work object WA. The distance detection device 20 obtains information on the distance from a predetermined position of the hydraulic excavator 1 to the work target WA. Antennas 21 and 22 receive radio waves from positioning satellite 200. The global position calculation device 23 uses the radio waves received by the antennas 21 and 22 to determine the global positions of the antennas 21 and 22, that is, the positions in global coordinates.

<Overall configuration of hydraulic excavator 1>
FIG. 2 is a perspective view showing a hydraulic excavator 1, which is a working machine according to an embodiment. The hydraulic excavator 1 includes a vehicle main body 1B as a main body portion and a working machine 2. The vehicle body 1B includes an upper revolving structure 3 which is a revolving structure and a traveling device 5 which is a traveling object. The upper revolving body 3 accommodates devices such as an engine, which is a power generating device, and a hydraulic pump inside an engine room 3EG. In the embodiment, the hydraulic excavator 1 uses an internal combustion engine such as a diesel engine as the power generating device, but the power generating device is not limited to the internal combustion engine. The power generation device of the hydraulic excavator 1 may be, for example, a so-called hybrid type device that combines an internal combustion engine, a generator motor, and a power storage device. Further, the power generating device of the hydraulic excavator 1 may not include an internal combustion engine, but may be a device that combines a power storage device and a generator motor.

The upper revolving body 3 has a driver's cab 4 . The operator's cab 4 is installed on the other end side of the upper revolving structure 3. That is, the driver's cab 4 is installed on the opposite side to the side where the engine room 3EG is arranged. A handrail 9 is attached above the revolving upper structure 3.

The traveling device 5 is equipped with the upper revolving body 3 . The traveling device 5 has tracks 5a and 5b. The traveling device 5 is driven by one or both of hydraulic motors 5c provided on the left and right sides. The hydraulic excavator 1 is caused to travel by rotating crawler tracks 5a and 5b of the traveling device 5. The working machine 2 is attached to the side of the operator's cab 4 of the upper revolving body 3.

The hydraulic excavator 1 may be equipped with tires instead of the tracks 5a and 5b, and a traveling device that can travel by transmitting the driving force of the engine to the tires via a transmission. An example of such a hydraulic excavator 1 is a wheeled hydraulic excavator. Further, the hydraulic excavator 1 is equipped with a traveling device having such tires, a working machine is attached to the vehicle body (body part), and is equipped with an upper revolving structure 3 and its rotation mechanism as shown in FIG. For example, it may be a backhoe loader, which has a non-standard structure. That is, the backhoe loader is equipped with a working machine attached to a vehicle body and a traveling device forming a part of the vehicle body.

The upper revolving structure 3 has a front side where the work equipment 2 and the operator's cab 4 are arranged, and a rear side where the engine room 3EG is arranged. The longitudinal direction of the upper revolving body 3 is the x direction. The left side when facing forward is the left side of the upper revolving structure 3, and the right side when facing forward is the right side of the upper revolving structure 3. The left and right direction of the upper revolving structure 3 is also referred to as the width direction or the y direction. In the hydraulic excavator 1 or the vehicle main body 1B, the traveling device 5 side is at the bottom with respect to the upper revolving structure 3, and the upper revolving structure 3 side is upper with respect to the traveling device 5. The vertical direction of the upper revolving body 3 is the z direction. When the hydraulic excavator 1 is installed on a horizontal surface, the lower side is the vertical direction, that is, the side in which gravity acts, and the upper side is the side opposite to the vertical direction.

The work machine 2 includes a boom 6, an arm 7, a bucket 8 as a work tool, a boom cylinder 10, an arm cylinder 11, and a bucket cylinder 12. A base end portion of the boom 6 is rotatably attached to the front portion of the vehicle body 1B via a boom pin 13. The base end of the arm 7 is rotatably attached to the tip of the boom 6 via an arm pin 14. A bucket 8 is attached to the tip of the arm 7 via a bucket pin 15. The bucket 8 rotates around the bucket pin 15. A plurality of blades 8B are attached to the bucket 8 on the opposite side of the bucket pin 15. The cutting edge 8T is the tip of the blade 8B.

The bucket 8 does not need to have a plurality of blades 8B. That is, the bucket may not have the blade 8B as shown in FIG. 2, but may have a straight blade edge formed of a steel plate. The work machine 2 may include, for example, a tilt bucket having a single blade. A tilt bucket is equipped with a bucket tilt cylinder, and by tilting the bucket left and right, even when the hydraulic excavator is on a slope, it can shape slopes and flat land into any shape and level the ground. This bucket can also perform compaction work using plates. In addition to this, the work machine 2 may include a slope bucket or a rock drilling attachment equipped with a rock drilling chip as a working tool instead of the bucket 8.

The boom cylinder 10, arm cylinder 11, and bucket cylinder 12 shown in FIG. 2 are hydraulic cylinders each driven by the pressure of hydraulic oil discharged from a hydraulic pump. The boom cylinder 10 drives the boom 6 to raise and lower it. The arm cylinder 11 drives the arm 7 to rotate around the arm pin 14. The bucket cylinder 12 drives the bucket 8 to rotate around the bucket pin 15.

Antennas 21, 22 and an antenna 24 are attached to the upper part of the upper revolving body 3. The antennas 21 and 22 are used to detect the current position of the hydraulic excavator 1. The antennas 21 and 22 are electrically connected to a global position calculation device 23 shown in FIG. The global position calculation device 23 is a position detection device that detects the position of the hydraulic excavator 1. The global position calculation device 23 detects the current position of the hydraulic excavator 1 using RTK-GNSS (Real Time Kinematic - Global Navigation Satellite Systems, GNSS refers to a global navigation satellite system). In the following description, the antennas 21 and 22 will be appropriately referred to as GNSS antennas 21 and 22. Signals corresponding to the GNSS radio waves received by the GNSS antennas 21 and 22 are input to the global position calculation device 23. The global position calculation device 23 determines the installation positions of the GNSS antennas 21 and 22 in the global coordinate system. An example of a global navigation satellite system is GPS (Global Positioning System), but the global navigation satellite system is not limited to this.

As shown in FIG. 2, the GNSS antennas 21 and 22 are preferably installed on the upper revolving body 3 at both ends of the hydraulic excavator 1 separated from each other in the left-right direction, that is, in the width direction. In the embodiment, the GNSS antennas 21 and 22 are attached to handrails 9 attached to both sides of the upper revolving body 3 in the width direction, respectively. The position where the GNSS antennas 21 and 22 are attached to the upper revolving body 3 is not limited to the handrail 9, but it is better to install the GNSS antennas 21 and 22 as far away as possible from the hydraulic excavator 1. This is preferable because the detection accuracy of the current position is improved. Furthermore, it is preferable that the GNSS antennas 21 and 22 be installed at positions that do not obstruct the operator's field of view as much as possible.

The imaging device 19 images the work object WA shown in FIG. 1, and the distance detection device 20 calculates the distance from itself (a predetermined position of the hydraulic excavator 1) to the work object WA, so that the work object WA is as wide as possible. It is preferable to obtain information from. For this reason, in the embodiment, the antenna 24, the imaging device 19, and the distance detection device 20 are installed above the driver's cab 4 of the revolving upper structure 3. The location where the imaging device 19 and the distance detection device 20 are installed is not limited to above the driver's seat 4. For example, the imaging device 19 and the distance detection device 20 may be installed inside and above the driver's cab 4.

In the imaging device 19, an imaging surface 19L faces the front of the upper rotating body 3. The distance detection device 20 has a detection surface 20L facing forward of the upper revolving structure 3. In the embodiment, the imaging device 19 is a monocular camera equipped with an image sensor such as a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor. In embodiments, the distance detection device 20 is a three-dimensional laser range finder or a distance sensor. The imaging device 19 and the distance detection device 20 are not limited to these. For example, instead of the imaging device 19 and the distance detection device 20, a device having both the function of acquiring an image of the work target WA and the function of determining the distance to the work target WA may be used. An example of such a device is a stereo camera.

<Control system of hydraulic excavator 1>
FIG. 3 is a diagram showing a control system 1S of the hydraulic excavator 1, which is a working machine according to the embodiment. The control system 1S includes a communication device 25, a sensor controller 26, a work equipment control device 27, an imaging device 19, a distance detection device 20, a global position calculation device 23, an attitude detection device 32, and an IMU (Inertial Measurement Unit: inertial measurement device) 33 and a hydraulic system 36. The communication device 25, the sensor controller 26, and the work machine control device 27 are connected by a signal line 35. With this structure, the communication device 25, the sensor controller 26, and the work implement control device 27 can exchange information with each other via the signal line 35. An example of a signal line that transmits information within the control system 1S is an in-vehicle signal line such as a CAN (Controller Area Network).

The sensor controller 26 includes a processor such as a CPU (Central Processing Unit), and a storage device such as a RAM and a ROM. The sensor controller 26 receives input values detected by the global position calculation device 23, information on images captured by the imaging device 19, detected values by the distance detection device 20, detected values by the attitude detection device 32, and detected values by the IMU 33. . The sensor controller 26 transmits the input detection values and image information to the processing device 51 of the facility 50 shown in FIG. 1 via the signal line 35 and the communication device 25.

The work machine control device 27 includes a processor such as a CPU (Central Processing Unit), and a storage device such as a RAM (Random Access Memory) and a ROM (Read Only Memory). The work machine control device 27 acquires, via the communication device 25, a command for operating at least one of the work machine 2 and the revolving upper structure 3, which is generated by the processing device 51 of the facility 50. The work machine control device 27 controls the electromagnetic control valve 28 of the hydraulic system 36 based on the acquired command.

The hydraulic system 36 includes an electromagnetic control valve 28, a hydraulic pump 29, and hydraulic actuators such as a boom cylinder 10, an arm cylinder 11, a bucket cylinder 12, and a swing motor 30. The hydraulic pump 29 is driven by the engine 31 and discharges hydraulic oil for operating the hydraulic actuator. The work equipment control device 27 controls the flow rate of hydraulic oil supplied to the boom cylinder 10 , arm cylinder 11 , bucket cylinder 12 , and swing motor 30 by controlling the electromagnetic control valve 28 . In this way, the work equipment control device 27 controls the operations of the boom cylinder 10, arm cylinder 11, bucket cylinder 12, and swing motor 30.

The sensor controller 26 acquires detection values of the first stroke sensor 16 , the second stroke sensor 17 , and the third stroke sensor 18 . The first stroke sensor 16 is provided on the boom cylinder 10, the second stroke sensor 17 is provided on the arm cylinder 11, and the third stroke sensor 18 is provided on the bucket cylinder 12.

The first stroke sensor 16 detects a boom cylinder length, which is the length of the boom cylinder 10, and outputs it to the sensor controller 26. The second stroke sensor 17 detects the arm cylinder length, which is the length of the arm cylinder 11, and outputs it to the sensor controller 26. The third stroke sensor 18 detects the bucket cylinder length, which is the length of the bucket cylinder 12, and outputs it to the sensor controller 26.

Once the boom cylinder length, arm cylinder length, and bucket cylinder length are determined, the attitude of the working machine 2 is determined. Therefore, the first stroke sensor 16, second stroke sensor 17, and third stroke sensor 18 that detect these correspond to the posture detection device 32 that detects the posture of the working machine 2. The posture detection device 32 is not limited to the first stroke sensor 16, the second stroke sensor 17, and the third stroke sensor 18, and may be an angle detector.

The sensor controller 26 calculates the inclination angle of the boom 6 with respect to the direction (z-axis direction) orthogonal to the horizontal plane in the local coordinate system, which is the coordinate system of the hydraulic excavator 1, from the boom cylinder length detected by the first stroke sensor 16. The work equipment control device 27 calculates the inclination angle of the arm 7 with respect to the boom 6 from the arm cylinder length detected by the second stroke sensor 17. The work machine control device 27 calculates the inclination angle of the bucket 8 with respect to the arm 7 from the bucket cylinder length detected by the third stroke sensor 18. The inclination angles of the boom 6, arm 7, and bucket 8 are information indicating the attitude of the working machine 2. That is, the sensor controller 26 obtains information indicating the attitude of the working machine 2. The sensor controller 26 transmits the calculated inclination angle to the processing device 51 of the facility 50 shown in FIG. 1 via the signal line 35 and the communication device 25.

The GNSS antenna 21 receives a position P1 indicating its own position from a positioning satellite. The GNSS antenna 22 receives a position P2 indicating its own position from a positioning satellite. The GNSS antennas 21 and 22 receive positions P1 and P2 at a frequency of, for example, 10 Hz. The positions P1 and P2 are information on the positions where the GNSS antennas are installed in the global coordinate system. Signals corresponding to the GNSS radio waves received by the GNSS antennas 21 and 22, that is, the positions P1 and P2, are input to the global position calculation device 23. The GNSS antennas 21 and 22 output the positions P1 and P2 to the global position calculation device 23 every time they receive the positions P1 and P2.

The global position calculation device 23 includes a processor such as a CPU, and a storage device such as a RAM and a ROM. The global position calculation device 23 detects the positions P1 and P2 of the GNSS antennas 21 and 22 in the global coordinate system at a frequency of, for example, 10 Hz, and outputs them to the sensor controller 26 as reference position information Pga1 and Pga2. In the embodiment, the global position calculation device 23 calculates the azimuth angle of the hydraulic excavator 1, more specifically the yaw angle which is the azimuth angle of the upper revolving structure 3, from the acquired two positions P1 and P2, and calculates the yaw angle which is the azimuth angle of the upper rotating structure 3, and calculates the yaw angle which is the azimuth angle of the upper rotating structure 3. Output to. The sensor controller 26 transmits the acquired reference position information Pga1, Pga2 and yaw angle to the processing device 51 of the facility 50 shown in FIG. 1 via the signal line 35 and the communication device 25.

The IMU 33 detects the operation and posture of the hydraulic excavator 1. The operation of the hydraulic excavator 1 includes at least one of the operation of the upper revolving structure 3 and the operation of the traveling device 5. The posture of the hydraulic excavator 1 can be expressed by the roll angle, pitch angle, and yaw angle of the hydraulic excavator 1. In the embodiment, the IMU 33 detects and outputs the angular velocity and acceleration of the hydraulic excavator 1.

<About the coordinate system>
FIG. 4 is a diagram for explaining coordinate systems in the image display system 100 and the remote control system 101 according to the embodiment. FIG. 5 is a rear view of the hydraulic excavator 1. FIG. 6 is a diagram illustrating the coordinate systems of the imaging device and the distance detection device. In the image display system 100 and the remote control system 101, there are a global coordinate system, a local coordinate system, a coordinate system of the imaging device 19, and a coordinate system of the distance detection device 20. In the embodiment, the global coordinate system is, for example, a coordinate system in GNSS. The global coordinate system is a three-dimensional coordinate system indicated by (X, Y, Z) based on, for example, a reference position PG of a reference pile 80, which is a reference installed in the work area GA of the hydraulic excavator 1. As shown in FIG. 5, the reference position PG is located, for example, at the tip 80T of the reference pile 80 installed in the work area GA.

The local coordinate system is a three-dimensional coordinate system indicated by (x, y, z) with the hydraulic excavator 1 as a reference. In the embodiment, the origin position PL of the local coordinate system is the intersection of the z-axis, which is the rotation center axis of the upper revolving body 3, and a plane perpendicular to the z-axis within the swing circle of the upper revolving body 3. It is not limited. The plane perpendicular to the z-axis within the swing circle can be a plane passing through the center of the swing circle in the z-axis direction.

In the embodiment, the coordinate system of the imaging device 19 is a three-dimensional coordinate system indicated by (Xc, Yc, Zc) with the center of the light receiving surface 19P of the imaging device 19RC as the origin PC, as shown in FIG. be. In the embodiment, as shown in FIG. 6, the coordinate system of the distance detection device 20 is a three-dimensional coordinate system indicated by (Xd, Yd, Zd) with the center of the light receiving surface 20P of the distance detection element 20RC as the origin PD. It is a system.

<Posture of hydraulic excavator 1>
As shown in FIG. 5, the inclination angle θ4 of the upper revolving structure 3 in the left-right direction, that is, the width direction, is the roll angle of the hydraulic excavator 1, and the inclination angle θ5 of the upper revolving structure 3 in the longitudinal direction is the pitch of the hydraulic excavator 1. The angle of the upper rotating body 3 around the z-axis is the yaw angle of the hydraulic excavator 1. The roll angle was detected by integrating the angular velocity around the x-axis detected by the IMU 33 over time, the pitch angle was detected by integrating the angular velocity around the y-axis detected by the IMU 33 over time, and the yaw angle was detected by the IMU 33. It is obtained by integrating the angular velocity around the z-axis over time. The angular velocity around the z-axis is the turning angular velocity ω of the hydraulic excavator 1. That is, by integrating the turning angular velocity ω over time, the yaw angle of the hydraulic excavator 1, more specifically, the upper rotating structure 3, can be obtained.

The acceleration and angular velocity detected by the IMU 33 are output to the sensor controller 26 as operation information. The sensor controller 26 performs processing such as filtering and integration on the operation information acquired from the IMU 33 to obtain a tilt angle θ4 which is a roll angle, a tilt angle θ5 which is a pitch angle, and a yaw angle. The sensor controller 26 transmits the obtained inclination angle θ4, inclination angle θ5, and yaw angle as information related to the attitude of the hydraulic excavator 1 as shown in FIG. 1 via the signal line 35 and the communication device 25 shown in FIG. The information is transmitted to the processing device 51 of the facility 50 where the data is stored.

The sensor controller 26 obtains information indicating the attitude of the working machine 2, as described above. Specifically, the information indicating the attitude of the work equipment 2 includes the inclination angle θ1 of the boom 6 with respect to the direction perpendicular to the horizontal plane (z-axis direction) in the local coordinate system, the inclination angle θ2 of the arm 7 with respect to the boom 6, and the inclination angle θ2 with respect to the arm 7. This is the inclination angle θ3 of the bucket 8. The processing device 51 of the facility 50 shown in FIG. , appropriately referred to as the blade edge position) P4 is calculated.

The storage unit 51M of the processing device 51 stores data on the work machine 2 (hereinafter referred to as work machine data as appropriate). The work machine data includes the length L1 of the boom 6, the length L2 of the arm 7, and the length L3 of the bucket 8. As shown in FIG. 4, the length L1 of the boom 6 corresponds to the length from the boom pin 13 to the arm pin 14. The length L2 of the arm 7 corresponds to the length from the arm pin 14 to the bucket pin 15. The length L3 of the bucket 8 corresponds to the length from the bucket pin 15 to the cutting edge 8T of the bucket 8. The cutting edge 8T is the tip of the blade 8B shown in FIG. Further, the work machine data includes information on the position up to the boom pin 13 with respect to the origin position PL of the local coordinate system. The processing device 51 can determine the blade edge position P4 with respect to the origin position PL using the lengths L1, L2, L3, inclination angles θ1, θ2, θ3, and the origin position PL. In the embodiment, the processing device 51 of the facility 50 determines the blade edge position P4, but the sensor controller 26 of the hydraulic excavator 1 may determine the blade edge position P4 and transmit it to the processing device 51 of the facility 50.

<Example of control executed by image display system 100 and remote control system 101>
FIG. 7 is a flowchart of an example of control executed by the image display system 100 and the remote control system 101. FIG. 8 is a diagram showing the imaging device 19, the distance detection device 20, and the work target WA.

In step S101, the sensor controller 26 shown in FIG. 3 acquires information about the hydraulic excavator 1. Information about the hydraulic excavator 1 is information obtained from the imaging device 19, the distance detection device 20, the global position calculation device 23, the attitude detection device 32, and the IMU 33. As shown in FIG. 8, the imaging device 19 images the work target WA within the imaging range TA to obtain an image of the work target WA. The distance detection device 20 detects the distance Ld from the distance detection device 20 to the work target WA and other objects existing within the detection range MA. The global position calculation device 23 obtains reference position information Pga1, Pga2 corresponding to the positions P1, P2 of the GNSS antennas 21, 22 in the global coordinate system. The attitude detection device 32 detects a boom cylinder length, an arm cylinder length, and a bucket cylinder length. The IMU 33 detects the attitude of the hydraulic excavator 1, more specifically, the roll angle θ4, pitch angle θ5, and yaw angle of the upper revolving structure 3.

In step S102, the processing device 51 of the image display system 100 and the remote control system 101 receives the hydraulic pressure from the sensor controller 26 of the hydraulic excavator 1 via the communication device 25 of the hydraulic excavator 1 and the communication device 54 connected to the processing device 51. Obtain information about shovel 1.

The information on the hydraulic excavator 1 that the processing device 51 acquires from the sensor controller 26 includes an image of the work target WA captured by the imaging device 19 and a distance from the distance detection device 20 to the work target WA detected by the distance detection device 20. information on the attitude of the working machine 2 included in the hydraulic excavator 1 detected by the attitude detection device 32, reference position information Pga1, Pga2, and information on the attitude of the hydraulic excavator 1.

Information on the distance from the distance detection device 20 to the work target WA includes a distance Ld to the work target WA or object OB existing within the detection range MA, and information on the orientation of the position Pd corresponding to the distance Ld. In FIG. 8, the distance Ld is shown as the distance to the work target WA. The information on the orientation of the position Pd is the orientation of the position Pd with respect to the distance detection device 20, and is the angle with respect to each axis Xd, Yd, and Zd of the coordinate system of the distance detection device 20. The information on the posture of the work machine 2 acquired by the processing device 51 is the inclination angles θ1, θ2, and θ3 of the work machine 2, which are determined by the sensor controller 26 using the boom cylinder length, arm cylinder length, and bucket cylinder length. Information on the posture of the hydraulic excavator 1 is the roll angle θ4, pitch angle θ5, and yaw angle of the hydraulic excavator 1, more specifically, the upper revolving structure 3.

The processing device 51 uses the inclination angles θ1, θ2, and θ3 of the work equipment 2 acquired from the sensor controller 26, the length L1 of the boom 6, the length L2 of the arm 7, and the length of the bucket 8 stored in the storage unit 51M. The blade edge position P4 of the bucket 8 is determined using the distance L3. The blade edge position P4 of the bucket 8 is a set of coordinates in the local coordinate system (x, y, z) of the hydraulic excavator 1.

Proceeding to step S103, the processing device 51 converts the distance Ld to the work target WA into position information using the information on the distance to the work target WA. The position information is the coordinates of the position Pd in the coordinate system (Xd, Yd, Zd) of the distance detection device 20. In step S103, all distances Ld detected by the distance detection device 20 within the detection range MA are converted into position information. The processing device 51 converts the distance Ld into position information using the distance Ld and information on the direction of the position Pd corresponding to the distance Ld. In step S103, the distance to the object OB existing within the detection range MA is also converted into position information, similar to the distance Ld of the work target WA. Through the process in step S103, information on the position of the work target WA within the detection range MA is obtained. Information on the shape of the work target WA can be obtained from information on the position of the work target WA.

The position information and shape information of the work target WA are a set of coordinates of the position Pd in the coordinate system (Xd, Yd, Zd) of the distance detection device 20. The processing device 51 converts the information on the shape of the work target WA into values in the coordinate system (Xc, Yc, Zc) of the imaging device 19, and then converts the information into values in the local coordinate system (x, y, z) of the hydraulic excavator 1. Convert to

In step S104, the processing device 51 uses the information on the position of the work target WA, the blade edge position P4 of the bucket 8, and the reference position information Pga1 and Pga2 acquired from the sensor controller 26 of the hydraulic excavator 1 in the global coordinate system (X, Y, Z). Upon conversion to the global coordinate system (X, Y, Z), the processing device 51 generates a rotation matrix using the roll angle θ4, pitch angle θ5, and yaw angle of the hydraulic excavator 1 acquired from the sensor controller 26. The processing device 51 uses the generated rotation matrix to convert the information on the position of the work target WA, the blade edge position P4 of the bucket 8, and the reference position information Pga1, Pga2 into the global coordinate system (X, Y, Z). Next, the process proceeds to step S105, and the processing device 51 calculates an occupied area.

FIG. 9 is a diagram illustrating the occupied area SA. The occupied area SA is an area occupied by the work machine 2 within the information on the shape of the work object WA. In the example shown in FIG. 9, a part of the bucket 8 of the work machine 2 is within the detection range MA of the distance detection device 20 and between the distance detection device 20 and the work object WA. Therefore, in the occupied area SA, the distance detection device 20 detects the distance to the bucket 8 instead of the distance to the work target WA. In the embodiment, the processing device 51 removes the occupied area SA from the information on the shape of the work target WA obtained in step S103.

The processing device 51 stores information on at least one of the position and orientation detected by the distance detection device 20 in accordance with at least one of the position and orientation of the bucket 8, for example, in the storage unit 51M. Such information is included in the attitude of the working machine 2 of the hydraulic excavator 1 in this embodiment. The posture of the work equipment 2 is determined using the inclination angles θ1, θ2, θ3 of the work equipment 2, the length L1 of the boom 6, the length L2 of the arm 7, and the length L3 of the bucket 8, and the position of the hydraulic excavator 1 as necessary. It can be determined using posture. The processing device 51 then compares the data detected by the distance detection device 20 with the information stored in the storage unit 51M, and if the two match, it can be determined that the bucket 8 has been detected. . Through processing using the attitude of the work equipment 2, the processing device 51 does not use the information on the buckets 8 of the occupied area SA when generating the grid image 65 shown in FIG. Can be generated accurately.

In order to remove the portion of the occupied area SA, the process using the attitude of the working machine 2 may be performed by the following method. Information regarding at least one of the position and orientation of the bucket 8 in the global coordinate system, which is included in the attitude of the work equipment 2, includes the inclination angles θ1, θ2, θ3 of the work equipment 2, the length L1 of the boom 6, and the length of the arm 7. It is determined from L2 and the length L3 of the bucket 8. In steps S103 and S104, information on the shape of the work target WA in the global coordinate system is obtained. In step S106, the processing device 51 removes the area obtained by projecting the position of the bucket 8 onto the shape information of the work target WA as the occupied area SA from the shape of the work target WA.

FIG. 10 is a diagram showing information on the shape of the work target WA from which the occupied area has been removed. The information IMWA about the shape of the work target WA is a set of coordinates Pgd (X, Y, Z) in the global coordinate system (X, Y, Z). As for the occupied area IMBA, no coordinate information exists due to the process of step S106. Next, the process proceeds to step S107, and the processing device 51 generates an image indicating the position of the bucket 8. The image showing the position of the bucket 8 is an image of a portion of the work target WA that corresponds to the bucket 8.

11 to 13 are diagrams for explaining images showing the position of the bucket 8 on the work target WA. In the embodiment, the image showing the position of the bucket 8 is an image showing the position of the cutting edge 8T of the bucket 8 on the work object WA. In the following, an image showing the position of the blade edge 8T of the bucket 8 will be appropriately referred to as a blade edge position image. As shown in FIG. 11, the blade edge position image is based on the position Pgt (X, Y, Z ). The vertical direction is the Z direction in the global coordinate system (X, Y, Z), and is a direction perpendicular to the X direction and the Y direction.

As shown in FIG. 12, between the first position Pgt1 (X1, Y1, Z1) of the surface WAP of the work target WA and the second position Pgt2 (X2, Y2, Z2), the surface WAP of the work target WA is The line image formed along the line is the blade edge position image 61. The first position Pgt1 (X1, Y1, Z1) is a straight line LV1 extending vertically from a position Pgb1 on the outside of the blade 8B at one end 8Wt1 side in the width direction Wb of the bucket 8 and the surface of the work target WA. This is the intersection with WAP. The second position Pgt2 (X2, Y2, Z2) is a straight line LV2 extending vertically from a position Pgb2 outside the blade 8B on the other end 8Wt2 side of the bucket 8 in the width direction Wb and the surface of the work target WA. This is the intersection with WAP. The width direction Wb of the bucket 8 is the direction in which the plurality of blades 8B are arranged.

The processing device 51 determines a straight line LV1 and a straight line LV2 extending in the vertical direction from the positions Pgb1 and Pgb2 of the bucket 8. Next, the processing device 51 calculates a first position Pgt1 (X1, Y1, Z1) and a second position Pgt2 (X2, Y2, Z2) from the obtained straight line LV1 and straight line LV2 and the information on the shape of the work target WA. ). Then, the processing device 51 sets a set of positions Pgt of the surface WAP of the work object WA when a straight line connecting the first position Pgt1 and the second position Pgt2 is projected onto the surface WAP of the work object WA as a blade edge position image 61.

In the embodiment, the processing device 51 generates a first straight line image 62 that is an image of a straight line LV1 connecting the position Pgb1 and the first position Pgt1 (X1, Y1, Z1), and a first straight line image 62 that is an image of the straight line LV1 connecting the position Pgb1 and the first position Pgt1 (X1, Y1, Z1), , Z2), which is an image of the straight line LV2, is generated. Next, the processing device 51 converts the blade edge position image 61, the first straight line image 62, and the second straight line image 63 into images with the imaging device 19 as a reference, that is, images from the viewpoint of the imaging device 19.

As shown in FIG. 13, the image from the viewpoint of the imaging device 19 is obtained from the origin Pgc (Xc, Yc, Zc) of the imaging device in the global coordinate system (X, Y, Z) to the blade edge position image 61 and the first straight line image. 62 and a second straight line image 63. The origin Pgc (Xc, Yc, Zc) of the imaging device is the coordinate obtained by converting the center of the light receiving surface 19P of the imaging element 19RC included in the imaging device 19, that is, the origin PC, into the global coordinate system (X, Y, Z). .

The blade edge position image 61, the first straight line image 62, and the second straight line image 63 are images in a three-dimensional space, but the image from the viewpoint of the imaging device 19 is a two-dimensional image. Therefore, the processing device 51 converts the blade edge position image 61, first straight line image 62, and second straight line image 63 defined in a three-dimensional space, that is, the global coordinate system (X, Y, Z) onto a two-dimensional plane. Performs a perspective transformation to project. The blade edge position image 61, the first straight line image 62, and the second straight line image 63 that have been converted into images from the viewpoint of the imaging device 19 are hereinafter appropriately referred to as a work tool guide image 60.

14 and 15 are diagrams showing a grid image 65 that is a reference image. Once the work implement guide image 60 is generated, the process proceeds to step S108, and the processing device 51 generates a grid image 65 that is a reference image. The grid image 65 is a line image along the surface WAP of the work target WA using information on the position of the work target WA. The grid image 65 is a grid including a plurality of first line images 66 and a plurality of second line images 67 that intersect with the plurality of first line images 66. In the embodiment, the first line image 66 is, for example, a line image extending parallel to the X direction and arranged in the Y direction in the global coordinate system (X, Y, Z). The first line image 66 is a line image extending parallel to the front-rear direction of the upper revolving structure 3 provided in the hydraulic excavator 1 and arranged in the width direction of the upper revolving structure 3 in the global coordinate system (X, Y, Z). It may be.

The grid image 65 is generated using information on the position of the work target WA, more specifically, the position Pgg (X, Y, Z) of the front surface WAP. The intersection of the first line image 66 and the second line image 67 is the position Pgg (X, Y, Z). As shown in FIG. 15, the first line image 66 and the second line image 67 are defined in the global coordinate system (X, Y, Z) and therefore contain three-dimensional information. In the embodiment, the plurality of first line images 66 are arranged at equal intervals, and the plurality of second line images 67 are arranged at equal intervals. The interval between adjacent first line images 66 and the interval between adjacent second line images 67 are equal.

The grid image 65 is an image obtained by converting a first line image 66 and a second line image 67 generated using the position Pgg (X, Y, Z) of the front surface WAP into an image from the viewpoint of the imaging device 19. It is. After generating the first line image 66 and the second line image 67, the processing device 51 converts them into images from the viewpoint of the imaging device 19 to generate a grid image 65. By converting the first line image 66 and the second line image 67 into images from the viewpoint of the imaging device 19, grid images 65 equally spaced in the horizontal plane are created to assist in determining the absolute distance of the work target WA. It can be deformed and displayed according to the shape of the work target WA.

Next, in step S109, the processing device 51 removes the aforementioned occupied area SA from the generated work tool guide image 60 and the grid image 65 which is the reference image. In step S109, the processing device 51 converts the occupied area SA into an image from the viewpoint of the imaging device 19, and removes it from the work tool guide image 60 and the grid image 65, which is the reference image. In the embodiment, the processing device 51 generates a blade edge position image 61, a first straight line image 62, and a second straight line image 63 before being converted to an image from the viewpoint of the imaging device 19, and a first straight line image 62 and a second straight line image 63 which are converted to an image from the viewpoint of the imaging device 19. The occupied area SA before being converted into an image from the viewpoint of the imaging device 19 may be removed from the first line image 66 and the second line image 67 before being converted, respectively.

FIG. 16 is a diagram showing a work image 69. In step S110, the processing device 51 combines the work tool guide image 60 from which the occupied area SA has been removed, the grid image 65, and the image 68 of the work object WA captured by the imaging device 19, and Image 69 is generated. In step S111, the processing device 51 displays the generated work image 68 on the display device 52. The work image 69 is an image in which a grid image 65 and a work tool guide image 60 are displayed on the image 68 of the work target WA.

Since the grid image 65 is a grid along the surface WAP of the work object WA, the operator of the hydraulic excavator 1 can grasp the position of the work object WA by referring to the grid image 65. For example, the operator can use the second line image 67 to grasp the depth, that is, the longitudinal position of the upper revolving body 3 of the hydraulic excavator 1, and the first line image 66 to grasp the position of the bucket 8 in the width direction. .

In the work tool guide image 60, a blade edge position image 61 is displayed along the surface WAP of the work object WA and a grid image 65. Therefore, the operator can grasp the positional relationship between the bucket 8 and the work target WA from the grid image 65 and the blade edge position image 61, thereby improving work efficiency and work accuracy. In the embodiment, a first straight line image 62 and a second straight line image 63 connect both ends of the blade edge position image 61 from both sides of the bucket 8 in the width direction Wb. The operator can more easily grasp the positional relationship between the bucket 8 and the work object WA using the first straight line image 62 and the second straight line image 63. Since the grid image 65 and the blade edge position image 61 are displayed along the shape of the terrain to be worked on (work target WA), the relative positional relationship between the two on the terrain surface (two-dimensional) can be easily grasped. . Furthermore, since the first line image 66 and the second line image 67 that make up the grid image 65 are arranged at equal intervals in the global coordinate system, it is easy to grasp the sense of distance on the topographic surface, and It becomes easier to understand the feeling.

In the embodiment, the work image 69 includes information 64 indicating the distance between the cutting edge 8T of the bucket 8 and the work target WA. This has the advantage that the operator can grasp the actual distance between the cutting edge 8T of the bucket 8 and the work target WA. The distance between the cutting edge 8T of the bucket 8 and the work target WA can be the distance from the cutting edge 8T at the center of the bucket 8 in the width direction Wb to the surface WAP of the work target WA.

Note that the information 64 includes, instead of or in addition to the distance between the cutting edge 8T of the bucket 8 and the work object WA, information regarding the posture such as the angle of the bucket 8, and information indicating the relative distance between the bucket 8 and the work object WA. , information indicating the relationship between, for example, the direction of the cutting edge 8T of the bucket 8 and the direction of the surface of the work target WA, information indicating the position of the bucket 8 in coordinates, information indicating the direction of the surface of the work target WA, and from the imaging device 19. Any spatial position information regarding the work tool or the work object W may be used as long as it includes information such as information indicating the distance in the x direction in the local coordinate system to the cutting edge 8T of the bucket 8.

That is, the processing device 51 determines the position of the bucket 8, which is a working tool, the attitude of the bucket 8, the position of the work object WA, the relative attitude of the work object WA, the relative distance between the bucket 8 and the work object WA, the bucket At least one of the relative postures between 8 and the work object WA may be determined and displayed on the display device 52.

As described above, the image display system 100 and the remote control system 101 overlap the work tool guide image 60 and the grid image 65 generated from the viewpoint of the imaging device 19 with the image 68 of the actual work target WA captured by the imaging device 19. The information is also displayed on the display device 52. Through such processing, the image display system 100 and the remote control system 101 allow the operator who remotely operates the hydraulic excavator 1 to use the image of the work target WA displayed on the display device 52 to inform the operator of the position of the bucket 8 and the work target WA. Since it is possible to easily understand the positional relationship between Even an inexperienced operator can easily grasp the positional relationship between the position of the bucket 8 and the work target WA by using the image display system 100 and the remote control system 101. As a result, deterioration in work efficiency and work accuracy is suppressed. In addition, the image display system 100 and the remote control system 101 display the work tool guide image 60, the grid image 65, and the image 68 of the actual work target WA on the display device 52 in a superimposed manner. By allowing the operator to focus on a single screen, work efficiency can be improved.

In the grid image 65, the interval between adjacent first line images 66 is equal to the interval between adjacent second line images 67. Therefore, by superimposing and displaying the grid image 65 and the image 68 of the actual work target WA captured by the imaging device 19, it becomes easier to grasp the work point on the work target WA. Moreover, by superimposing the blade edge position image 61 of the work tool guide image 60 and the grid image 65, the operator can easily grasp the distance traveled by the bucket 8, and thus work efficiency is improved.

In the work tool guide image 60 and the grid image 65, the occupied area SA, which is the area of the work machine 2, is removed. It is possible to avoid displaying the guide image 60 and the grid image 65 in an overlapping manner. As a result, the image display system 100 and the remote control system 101 can display the work image 69 on the display device 52 in a format that is easy for the operator to view.

In the embodiment, the work tool guide image 60 only needs to include at least the blade edge position image 61. The grid image 65 only needs to include at least a plurality of second line images 67, that is, a plurality of line images indicating a direction orthogonal to the longitudinal direction of the upper revolving structure 3 included in the hydraulic excavator 1. Further, the processing device 51 may change, for example, the color of the blade edge position image 61 of the work implement guide image 60 depending on the distance between the blade edge 8T of the bucket 8 and the work target WA. By doing so, the operator can easily grasp the distance between the position of the bucket 8 and the work target WA.

In the embodiment, the processing device 51 converts the information on the shape of the work target WA into the global coordinate system (X, Y, Z) to generate the work tool guide image 60 and the grid image 65. It is not necessary to convert the information in the global coordinate system (X, Y, Z). In this case, the processing device 51 handles information on the shape of the work target WA in the local coordinate system (x, y, z) of the hydraulic excavator 1, and generates the work tool guide image 60 and the grid image 65. When handling information on the shape of the work target WA in the local coordinate system (x, y, z) of the hydraulic excavator 1, the GNSS antennas 21 and 22 and the global position calculation device 23 are not necessary.

In the embodiment described above, a part of the hydraulic excavator 1 (for example, the bucket 8 as described above) detected by the distance detection device 20 is removed to obtain information on the shape of the work target WA (three-dimensional terrain data). However, the three-dimensional terrain data acquired in the past (for example, several seconds ago) is stored in the storage unit 51M of the processing device 51, and the processing unit 51P of the processing device 51 stores the current work target WA and its stored data. It is determined whether the three-dimensional topographic data and the three-dimensional topographic data are at the same position, and if they are the same position, the grid image 65 may be displayed using the past three-dimensional topographic data. That is, the processing device 51 can display the grid image 65 even if there is a terrain hidden by a part of the hydraulic excavator 1 when viewed from the imaging device 19, if past three-dimensional terrain data is available.

Further, instead of displaying the grid image 65 using a grid, for example, the grid image 65 may be displayed using the local coordinate system as a polar coordinate system. Specifically, concentric circles at equal intervals according to the distance from the center of the hydraulic excavator 1 (for example, the rotation center of the upper revolving structure 3) are drawn as a line image (second line image), and the upper revolving structure 3 is drawn as a line image (second line image). Radial line images (first line images) may be drawn at equal intervals from the center of rotation depending on the rotation angle. In this case, the second line image, which is a concentric line image, and the first line image, which is a radial line image from the center of rotation, intersect. By displaying such a grid image, it is possible to easily grasp the positional relationship between the position of the bucket 8 and the work target WA during turning or excavation.

<Blade tip position image 61 of loading shovel type working machine 2a>
FIG. 17 is a diagram for explaining a blade edge position image 61 when a loading shovel type work machine 2a is used. In the loading shovel, the bucket 8 rotates from the rear to the front of the hydraulic excavator 1, so that earth and sand can be scooped up. In the loading shovel type work machine 2a, the cutting edge 8T of the bucket 8 faces forward of the revolving upper structure 3, and excavates a work object WA, which is a work object in front of the revolving upper structure 3. In this case, as shown in FIG. 17, the blade edge position image 61 is based on the surface WAP of the work object WA when the blade edge 8T is projected onto the work object WA in the horizontal direction, that is, in the direction orthogonal to the direction in which gravity acts. This is an image defined by the position Pgt (X, Y, Z). The horizontal direction is the X direction or Y direction in the global coordinate system (X, Y, Z), and is a direction perpendicular to Z. The processing device 51 uses information on the position Pgt (X, Y, Z) of the surface WAP of the work target WA to create a blade edge position image 61, a first straight line image 62, and a second straight line image 62 using a method similar to the method described above. A straight line image 63 is generated. The processing device 51 converts the generated blade edge position image 61, first straight line image 62, and second straight line image 63 into an image from the viewpoint of the imaging device 19 to obtain a work tool guide image 60.

<About the process to obtain the blade edge position image 61>
FIG. 18 is a diagram illustrating a first modified example of the process for obtaining a blade edge position image. In the first modification, the processing device 51 determines a straight line 72 that is perpendicular to the intersection line 71 between the virtual plane 70 and the work object WA and that passes through the cutting edge 8T of the bucket 8. The virtual plane 70 is an xz plane in the local coordinate system (x, y, z) of the hydraulic excavator 1 shown in FIGS. 5 and 6. The xz plane passes through the center of the bucket 8 in the width direction Wb.

Next, the processing device 51 connects a straight line LV1 passing through the outer position Pgb1 of the blade 8B on one end 8Wt1 side of the bucket 8 in the width direction Wb and parallel to the straight line 72, and the other end in the width direction Wb. A straight line LV2 that passes through the outer position Pgb2 of the blade 8B on the 8Wt2 side and is parallel to the straight line 72 is determined. The intersection of the straight line LV1 and the surface WAP of the work object WA is a first position Pgt1, and the intersection of the straight line LV2 and the surface WAP of the work object WA is a second position Pgt2. The processing device 51 obtains a first position Pgt1 and a second position Pgt2, and calculates a set of positions Pgt of the surface WAP of the work object WA when a straight line connecting the first position Pgt1 and the second position Pgt2 is projected onto the surface WAP of the work object WA. Let it be a blade edge position image 61.

The first straight line image 62 and the second straight line image 63 are images of the straight line LV1 and the straight line LV2. The processing device 51 converts the generated blade edge position image 61, first straight line image 62, and second straight line image 63 into an image from the viewpoint of the imaging device 19 to obtain a work tool guide image 60. Since the bucket 8 moves parallel to the virtual plane 70, the blade edge position image 61 obtained by the process of the first modification shows the position where the blade edge 8T of the bucket 8 is directed toward the work object WA.

FIGS. 19 and 20 are diagrams for explaining a second modified example of the process for obtaining a blade edge position image. When the upper revolving body 3 of the hydraulic excavator 1 is tilted with respect to the horizontal plane, that is, the XY plane of the global coordinate system (X, Y, Z), as shown in FIG. The surface of WA may be inclined with respect to WAP. Since the bucket 8 moves parallel to the virtual plane 70 described above, when the cutting edge 8T is projected onto the surface WAP of the work object WA existing in the vertical direction of the cutting edge 8T to obtain the cutting edge position image 61, the moving direction of the bucket 8 is There is a possibility that a shift occurs between the blade edge position image 61 and the blade edge position image 61.

In the second modification, the processing device 51 determines a straight line LV1 and a straight line LV2 extending in the vertical direction from the blade edge position P4 of the bucket 8. Next, the processing device 51 rotates the obtained straight lines LV1 and LV2 by the angle at which the upper revolving body 3 of the hydraulic excavator 1 is inclined with respect to the horizontal plane, that is, the roll angle θ4. The direction in which the straight line LV1 and the straight line LV2 are rotated is the direction in which they become parallel to the virtual plane 70. In this case, the processing device 51 rotates the straight lines LV1 and LV2 by θ4 about the positions Pgb1 and Pgb2 of the bucket 8 on the plane PV12 formed by the straight lines LV1 and LV2. In this way, the processing device 51 obtains the rotated straight line LV1a and the straight line LV2a.

Next, the processing device 51 determines the intersection between the rotated straight line LV1a and the straight line LV2a and the surface WAP of the work object WA, and sets the obtained two intersections as a first position Pgt1a and a second position Pgt2a, respectively. . Then, the processing device 51 sets a set of positions Pgt of the surface WAP of the work object WA when a straight line connecting the first position Pgt1a and the second position Pgt2a is projected onto the surface WAP of the work object WA as a blade edge position image 61. The first straight line image 62 and the second straight line image 63 are images of the straight line LV1a and the straight line LV2a. The processing device 51 converts the generated blade edge position image 61, first straight line image 62, and second straight line image 63 into an image from the viewpoint of the imaging device 19 to obtain a work tool guide image 60. The blade edge position image 61 obtained by the process of the second modification shows the position where the blade edge 8T of the bucket 8 faces the work object WA.

<Modified example of control system of hydraulic excavator 1>
FIG. 21 is a diagram showing a control system 1Sa of a hydraulic excavator 1 according to a modification. The image display system 100 and remote control system 101 described above remotely controlled the hydraulic excavator 1 using the operating device 53 of the facility 50 shown in FIG. In this modification, a display device 52 is provided in the operator's cab 4 shown in FIG. 2, and a work image 69 is displayed on the display device 52 to assist the operator in the work of the hydraulic excavator 1. .

Therefore, in the control system 1Sa, a processing device 51 and an operating device 53a are connected to the signal line 35 of the control system 1S described above. A display device 52 is connected to the processing device 51 . The processing device 51 included in the control system 1Sa has the same functions as the processing device 51 included in the facility 50 shown in FIG. 1 in the image display system 100 and remote control system 101 described above. The display device 52 of the control system 1Sa may be a dedicated display device for displaying the work image 69, or may be a display device included in the hydraulic excavator 1. The operating device 53a is a device for operating the hydraulic excavator 1, and includes a left operating lever 53La and a right operating lever 53Ra. The operating device 53a may be of a pilot hydraulic type or an electric type.

The hydraulic excavator 1 equipped with the control system 1Sa transmits the work tool guide image 60 and grid image 65 generated from the viewpoint of the imaging device 19 to the operator's cab 4 along with the image 68 of the actual work target WA captured by the imaging device 19. It is displayed on the display device 52 inside. Through such processing, the hydraulic excavator 1 allows the operator operating the hydraulic excavator 1 to grasp the positional relationship between the position of the bucket 8 and the work target WA using the image of the work target WA displayed on the display device 52. It can be made easier. As a result, work efficiency and work accuracy can be improved. Further, even an inexperienced operator can easily grasp the positional relationship between the position of the bucket 8 and the work object WA by using the hydraulic excavator 1 equipped with the control system 1Sa. As a result, deterioration in work efficiency and work accuracy is suppressed. Furthermore, when working at night, even if it is difficult for the operator to visually see the actual work target WA, the operator can work while looking at the work tool guide image 60 and grid image 65 displayed on the display device 52. , a decrease in work efficiency is suppressed.

Although the embodiment has been described above, the embodiment is not limited to the content described above. Furthermore, the above-mentioned components include those that can be easily assumed by those skilled in the art, those that are substantially the same, and those that are in a so-called equivalent range. Furthermore, the aforementioned components can be combined as appropriate. Furthermore, at least one of various omissions, substitutions, and changes of components can be made without departing from the gist of the embodiments. The working machine is not limited to the hydraulic excavator 1, but may be another working machine such as a wheel loader or a bulldozer.

1 Hydraulic excavator 1B Vehicle body 1S, 1Sa Control system 2, 2a Work equipment 3 Upper revolving structure 4 Driver's seat 6 Boom 7 Arm 8 Bucket 8B Blade 8T Cutting edge 16 First stroke sensor 17 Second stroke sensor 18 Third stroke sensor 19 Imaging Device 20 Distance detection device 21, 22 Antenna (GNSS antenna)
23 Global position calculation device 26 Sensor controller 27 Work equipment control device 32 Attitude detection device 33 IMU
50 Facility 51 Processing device 52 Display device 53, 53a Operating device 60 Work tool guide image (image)
61 Blade edge position image 62 First line image 63 Second line image 65 Grid image 66 First line image 67 Second line image 68 Image 69 Work image 100 Work machine image display system (image display system)
101 Remote control system for working machines (remote control system)

Claims (13)

a measuring device that is attached to a working machine equipped with a working tool and has both a function of acquiring an image of a work target in front of the work machine and a function of acquiring the shape of the work target;
an attitude detection device that detects the attitude of the working machine;
Using information on the posture of the work equipment and information on the shape of the work object, an image of a portion of the work object corresponding to the work tool is generated, synthesized with the image of the work object, and displayed. A processing device for displaying on the device;
Image display systems for working machines, including: The processing device includes:
The working machine according to claim 1, wherein a line image along the surface of the work object is generated using information on the shape of the work object, and the line image is combined with an image of the work object and displayed on the display device. Image display system. The line image is
The image display system for a work machine according to claim 2, wherein the image display system is a grid including a plurality of first line images and a plurality of second line images intersecting the plurality of first line images. The working tool is a bucket,
The processing device includes:
According to any one of claims 1 to 3, a line image of a portion of the work object corresponding to the cutting edge of the bucket is generated as an image of a portion of the work object corresponding to the work tool. image display system for working machines. The processing device includes:
An image of a straight line connecting one end side in the width direction of the blade edge of the bucket and the work object and an image of a straight line connecting the other end side in the width direction of the blade edge of the bucket and the work object are generated. 5. The image display system for a working machine according to claim 4, wherein the image is combined with the image of the work object and displayed on a display device. The processing device includes:
The image display system for a working machine according to any one of claims 1 to 5, wherein spatial position information regarding the working implement or the working object is obtained and displayed on the display device. The processing device includes:
the position of the work tool, the posture of the work tool, the position of the work object, the relative posture of the work object, the relative distance between the work tool and the work object, and the distance between the work tool and the work object; The image display system for a working machine according to any one of claims 1 to 6, wherein at least one relative posture of the working machine is determined and displayed on the display device. The processing device includes:
Any one of claims 1 to 7, wherein the area occupied by the work implement in the image of the work object is determined using the attitude of the work implement, and the obtained area is removed from information on the shape of the work object. An image display system for a working machine according to item 1. The attitude detection device is attached to the work machine,
The processing device and the display device are provided in a facility equipped with an operating device for remotely controlling the working machine,
An image display system for a working machine according to any one of claims 1 to 8. When remotely controlling a working machine equipped with a working tool,
a display device;
Corresponding to the work tool on the work object using information on the posture of the work machine and information on the shape of the work object in front of the work machine acquired by a measuring device attached to the work machine. a processing device that generates an image of the portion based on the measuring device, synthesizes it with the image of the work target acquired by the measuring device, and displays the image on the display device;
Image display systems for working machines, including: An image display system for a working machine according to claim 10;
an operating device for operating the work machine included in the work machine;
remote control systems for working machines, including A working machine, comprising the image display system for a working machine according to any one of claims 1 to 9. Obtaining an image of a work target in front of a working machine equipped with a working machine having a working tool;
obtaining the shape of the work target;
Detecting the attitude of the working machine;
Using information on the posture of the work equipment and information on the shape of the work object, an image of a portion of the work object corresponding to the work tool is generated, synthesized with the image of the work object, and displayed. displaying it on the device; and
A method of displaying images of working machines, including:

Images