JP2021165526A - Work machine image display system - Google Patents

Abstract

To suppress reduction in work efficiency in working using a work machine comprising a work apparatus having a work tool.SOLUTION: A work machine image display system to be applied to a work machine comprising a work apparatus having a work tool and a revolving superstructure to which the work apparatus is attached includes: a position detection unit for detecting at least one of the work apparatus's posture and position; a distance detection device for calculating information on a distance from the work machine to a work target; and a processing device for creating a first image using information on the work tool's position acquired by the position detection unit and the information on the distance calculated by the distance detection device to make the first image be displayed on a display device, the first image being an image of part on the work target facing the work tool, the part extending in a revolving direction of the revolving superstructure and including part corresponding to the work tool's portion.SELECTED DRAWING: Figure 1

Description

The present invention relates to an image display system for a work machine.

As described in Patent Document 1, a technique for remotely controlling a work machine such as a hydraulic excavator is known.

Japanese Unexamined Patent Publication No. 2004-294067

When the work machine is remotely controlled, the displayed image is two-dimensional in the operation using the image from the operator's viewpoint of the work machine, so that the perspective is poor. Therefore, it becomes difficult to grasp the distance between the work target and the work machine, and the work efficiency may decrease. Also, when the operator on the work machine operates the work machine, it may be difficult to grasp the distance between the work machine and the work target depending on the skill level of the operator, which may reduce the work efficiency. ..

An aspect of the present invention is to suppress a decrease in work efficiency when working with a work machine equipped with a work machine having a work tool.

According to the first aspect, it is an image display system of a work machine applied to a work machine having a work tool and a work machine having a swivel body to which the work machine is attached, and detects the position of the work machine. The position detection unit, the distance detection device that obtains information on the distance from the work machine to the work target, the position information of the work tool obtained by the position detection unit, and the distance obtained by the distance detection device. The first image includes a processing device that generates a first image using the information on the position of the work target obtained from the information and displays the first image on the display device, and the first image is the work facing the work tool. An image display system of a work machine generated based on information on a first position corresponding to the work tool on the surface of a target and information on the position of a portion extending from the first position along a direction in which the swivel body turns. Is provided.

According to the second aspect, in the first aspect, in the first image, the distance between the turning center axis of the turning body and a part of the working tool is a radius, and the position corresponding to the turning center axis. An image display system for a working machine, which is an image of an arc centered on

According to the third aspect, in the second aspect, the work tool is a bucket, and a part of the work tool is an image of a work machine which is a cutting edge of the bucket located in the center in the width direction of the bucket. A display system is provided.

According to the fourth aspect, in any one of the first to third aspects, the processing apparatus uses the information on the position of the work object to produce a line image along the surface of the work object. Provided is an image display system of a work machine that is generated, combined with an image of the work target, and displayed on the display device.

According to the fifth aspect, in the fourth aspect, the line image is centered on a plurality of first line images radially extending from a position corresponding to the rotation center axis of the swivel body and the rotation center axis. Provided is an image display system of a working machine including a plurality of second line images extending along a turning direction of the swivel body.

According to the sixth aspect, in any one of the first to fifth aspects, the image pickup device attached to the swivel body is provided, and the processing device includes the first image and the image pickup device. Provided is an image display system of a work machine that synthesizes a second image, which is an image of the work object captured by the above, and displays the image on the display device.

According to the seventh aspect, in the sixth aspect, the processing apparatus uses the posture of the working machine to obtain an area occupied by the working machine in the second image, and the obtained area is used as the work target. An image display system for a working machine is provided that removes from the shape information of the work machine.

According to the eighth aspect, in the sixth or seventh aspect, the image pickup device, the position detection unit, and the distance detection device are provided in the work machine, and the processing device and the display device are described. An image display system for a work machine provided in a facility equipped with an operation device for remotely controlling the work machine is provided.

According to the ninth aspect, it is applied to a work machine having a work tool, a swivel body to which the work machine is attached, a position detection unit for detecting the position of the work machine, a distance detection device, and a work machine provided with an image pickup device. It is an image display system of a work machine, and information on the position of the display device and the work tool obtained by the position detection unit, and information on the distance from the work machine to the work target obtained by the distance detection device. A process of generating a first image using the information on the position of the work target obtained from the above, synthesizing it with a second image which is an image of the work target captured by the image pickup device, and displaying the image on the display device. The first image includes the device, and the information of the first position corresponding to the work tool on the surface of the work target facing the work tool, and the direction in which the swivel body turns from the first position. An image display system for a working machine is provided, which is generated based on information on the position of a portion extending along the line.

According to a tenth aspect, a remote control system for a work machine including an image display system for the work machine according to the ninth aspect and an operation device for operating the work machine included in the work machine is provided.

According to the eleventh aspect, the work machine provided with the image display system of the work machine according to any one of the first to ninth aspects is provided.

According to the twelfth aspect, it is applied to a work machine including a work tool, a work machine having the work tool, a swivel body to which the work machine is attached, and an imaging device attached to the swivel body. It is an image display method of the work machine to be performed, and the information of the position of the work tool obtained by using the posture of the work machine and the information of the distance from the work machine to the work target of the work target. Using the position information, information on the first position corresponding to a part of the work tool on the surface of the work object facing the work tool, and along the direction in which the swivel body turns from the first position. The first image includes a step of generating a first image based on information on the position of the extending portion and a step of displaying the generated first image on a display device, and the first image faces the work tool. Image display of the work machine generated based on the information of the first position corresponding to the work tool on the surface of the work target and the position information of the portion extending from the first position along the direction in which the swivel body turns. The method is provided.

INDUSTRIAL APPLICABILITY According to the present invention, it is possible to suppress a decrease in work efficiency when working with a work machine equipped with a work machine having a work tool.

FIG. 1 is a diagram showing an image display system of a work machine and a remote control system of the work machine according to the embodiment. FIG. 2 is a diagram showing a control system of a hydraulic excavator, which is a work machine according to an embodiment. FIG. 3 is a diagram for explaining the coordinate system of the image display system and the remote control system according to the embodiment. FIG. 4 is a rear view of the hydraulic excavator. FIG. 5 is a diagram illustrating a coordinate system of the image pickup device and the distance detection device. FIG. 6 is a flowchart of a control example executed by the image display system and the remote control system according to the embodiment. FIG. 7 is a diagram showing an image pickup device, a distance detection device, and a work target. FIG. 8 is a diagram illustrating an occupied area. FIG. 9 is a diagram showing information on the shape of the work target from which the occupied area has been removed. FIG. 10 is a diagram showing an example of the first image. FIG. 11 is a diagram for explaining a process in which the processing device generates the first image. FIG. 12 is a diagram for explaining a process in which the processing device generates the first image. FIG. 13 is a diagram showing the positional relationship between the origin of the image pickup apparatus and the first image, the first straight line image, and the second straight line image. FIG. 14 is a diagram showing a reference image and a first image shown in FIG. FIG. 15 is a diagram showing a working image. FIG. 16 is a diagram showing a reference image according to a modified example. FIG. 17 is a diagram showing an example of the first image and the third image. FIG. 18 is a diagram showing a control system of a hydraulic excavator according to a modified example.

An embodiment (embodiment) for carrying out the present invention will be described in detail with reference to the drawings.

<Overview of work machine image display system and work machine remote control system>
FIG. 1 is a diagram showing an image display system 100 of a work machine and a remote control system 101 of a work machine according to an embodiment. The image display system 100 of the work machine (hereinafter, appropriately referred to as the image display system 100) is a work target WA and a work tool of the hydraulic excavator 1 when the operator remotely operates the hydraulic excavator 1 which is a work machine. The bucket 8 and the bucket 8 are imaged by the image pickup device 19, and the obtained image is displayed on the display device 52. At this time, the image display system 100 displays a work image 69 including an image 60 for showing the position of the bucket 8 on the work target WA and an image 68 of the work target WA captured by the imaging device 19. Displayed on the device 52. Image 68 also includes an image of bucket 8. In an embodiment, the working image 69 further includes a reference image 65. The reference image 65 is an image displayed along the surface of the work target WA. The reference image 65 serves as an index showing the position of the work target WA. The work target of the hydraulic excavator 1 is a terrain surface that is a work target of the work machine 2 included in the hydraulic excavator 1, that is, a work target WA.

The image display system 100 includes an image pickup device 19, a posture detection device 32, a distance detection device 20, and a processing device 51. The remote control system 101 of the work machine (hereinafter, appropriately referred to as the remote control system 101) includes an image pickup device 19, an attitude detection device 32, a distance detection device 20, a work machine control device 27, and a display device 52. , A processing device 51, and an operating device 53. In the embodiment, the image pickup device 19, the posture detection device 32, and the distance detection device 20 of the image display system 100 are provided in the hydraulic excavator 1, and the processing device 51 is provided in the facility 50. The facility 50 is used for remotely controlling the hydraulic excavator 1 and managing the hydraulic excavator 1. In the embodiment, the image pickup device 19, the attitude detection device 32, the distance detection device 20, and the work equipment control device 27 of the remote control system 101 are provided in the hydraulic excavator 1, and the display device 52, the processing device 51, and the operation device 53 are facilities 50. Be prepared for.

The processing device 51 of the image display system 100 includes a processing unit 51P, a storage unit 51M, and an input / output unit 51IO. The processing unit 51P is exemplified by a processor such as a CPU (Central Processing Unit). Examples of the storage unit 51M include a RAM (Random Access Memory), a ROM (Read Only Memory), a hard disk drive, a storage device, or a combination thereof. The input / output unit 51IO is an interface circuit for connecting the processing device 51 and an external device. In the embodiment, a display device 52, an operation device 53, and a communication device 54 are connected to the input / output unit 51IO as external devices. The external device connected to the input / output unit 51IO is not limited to these.

The processing device 51 uses the position information of the bucket 8 obtained by using the posture of the work machine 2 and the position information of the work target WA obtained from the distance information obtained by the distance detection device 20. Based on the information of the first position corresponding to the bucket 8 on the work target WA facing the bucket 8 and the position information of the portion extending from the first position along the direction in which the upper swing body 3 which is a swing body turns. The generated first image is generated with reference to the image pickup apparatus 19. The first position may be a position corresponding to a part of the bucket 8 on the work target WA facing the bucket 8. The processing device 51 synthesizes the first image with the image 68 of the work target WA captured by the image pickup device 19 and displays it on the display device 52. The work target WA is a surface on which the work machine 2 of the hydraulic excavator 1 is to perform work such as excavation or leveling. The first image is included in the image 60 for showing the position of the bucket 8 on the work target WA. In the following, the image 68 is appropriately referred to as a second image 68.

Examples of the display device 52 include, but are not limited to, a liquid crystal display or a projector. The communication device 54 includes an antenna 54A. The communication device 54 communicates with the communication device 25 provided in the hydraulic excavator 1 to acquire information on the hydraulic excavator 1 or transmit information to the hydraulic excavator 1.

The operation device 53 has a left operation lever 53L installed on the left side of the operator and a right operation lever 53R arranged on the right side of the operator. The left operation lever 53L and the right operation lever 53R are compatible with two-axis operations for front / rear / left / right operations. For example, the operation of the right operating lever 53R in the front-rear direction corresponds to the operation of the boom 6 of the work machine 2 included in the hydraulic excavator 1. The operation of the right operation lever 53R in the left-right direction corresponds to the operation of the bucket 8 of the work machine 2. The operation of the left operation lever 53L in the front-rear direction corresponds to the operation of the arm 7 of the work machine 2. The operation of the left operating lever 53L in the left-right direction corresponds to the turning of the upper swing body 3 of the hydraulic excavator 1.

The amount of operation of the left operation lever 53L and the right operation lever 53R is detected by, for example, a potentiometer, a Hall IC, or the like, and the processing device 51 generates a control signal for controlling the electromagnetic control valve based on these detected values. .. This signal is sent to the work equipment control device 27 via the communication device 54 of the facility 50 and the communication device 25 of the hydraulic excavator 1. The work machine control device 27 controls the work machine 2 by controlling the electromagnetic control valve based on the control signal. The electromagnetic control valve will be described later.

The processing device 51 acquires an input for at least one of the left operating lever 53L and the right operating lever 53R, and generates an instruction for operating at least one of the working machine 2 and the upper swing body 3. The processing device 51 transmits the generated command to the communication device 25 of the hydraulic excavator 1 via the communication device 54. The work machine control device 27 included in the hydraulic excavator 1 acquires a command from the processing device 51 via the communication device 25, and operates at least one of the work machine 2 and the upper swivel body 3 according to the command.

The hydraulic excavator 1 includes a communication device 25, a work equipment control device 27, a posture detection device 32, an image pickup device 19, a distance detection device 20, antennas 21 and 22, and a position calculation device 23. The work machine control device 27 controls the work machine 2. The communication device 25 is connected to the antenna 24 and communicates with the communication device 54 provided in the facility 50. The work machine control device 27 controls the work machine 2 and the upper swing body 3. The posture detection device 32 detects the posture of at least one of the work machine 2 and the hydraulic excavator 1. The image pickup device 19 is attached to the hydraulic excavator 1 to take an image of the work target WA. The distance detection device 20 obtains information on the distance from the predetermined position of the hydraulic excavator 1 to the work target WA. The antennas 21 and 22 receive radio waves from the positioning satellite 200. The position calculation device 23 uses the radio waves received by the antennas 21 and 22 to obtain the global position of the antennas 21 and 22, that is, the position in the global coordinate system.

<Overall configuration of hydraulic excavator 1>
The hydraulic excavator 1 has a vehicle main body 1B and a working machine 2 as a main body portion. The vehicle body 1B has an upper turning body 3 and a traveling device 5 which is a traveling body. The upper swing body 3 has a driver's cab 4. The traveling device 5 is equipped with the upper swivel body 3. The traveling device 5 has a track. The traveling device 5 travels the hydraulic excavator 1 by rotating the crawler belt. The work machine 2 is attached to the side of the driver's cab 4 of the upper swing body 3.

In the upper swivel body 3, the side on which the work machine 2 and the driver's cab 4 are arranged is the front, and the side on which the counterweight 9 is arranged is the rear. The left side facing the front is the left side of the upper turning body 3, and the right side facing the front is the right side of the upper turning body 3. In the hydraulic excavator 1 or the vehicle body 1B, the traveling device 5 side is on the lower side with respect to the upper turning body 3, and the upper turning body 3 side is on the upper side with reference to the traveling device 5. When the hydraulic excavator 1 is installed on a horizontal plane, the lower part is in the vertical direction, that is, the side in which gravity acts, and the upper part is in the direction opposite to the vertical direction.

The working machine 2 has a boom 6, an arm 7, a bucket 8 which is a working tool, a boom cylinder 10, an arm cylinder 11, and a bucket cylinder 12. The bucket 8 has a plurality of blades 8B. The blade tip 8T is the tip of the blade 8B. The bucket 8 is not limited to the one having a plurality of blades 8B. The bucket 8 may be a tilt bucket. In addition to this, the working machine 2 may be provided with a slope bucket or an attachment for rock drilling having a tip for rock drilling as a working tool instead of the bucket 8.

The boom cylinder 10, the arm cylinder 11, and the bucket cylinder 12 are hydraulic cylinders that are driven by the pressure of hydraulic oil discharged from the hydraulic pump, respectively. The boom cylinder 10 drives the boom 6. The arm cylinder 11 drives the arm 7. The bucket cylinder 12 drives the bucket 8.

Antennas 21 and 22 and antennas 24 are attached to the upper part of the upper swing body 3. The antennas 21 and 22 are used to detect the position of the hydraulic excavator 1. The antennas 21 and 22 are electrically connected to the position arithmetic unit 23.

Antennas 21 and 22 are antennas for GNSS (Kinematic-Global Navigation Satellite Systems, where GNSS refers to global navigation satellite systems). The antennas 21 and 22 are arranged at a certain distance along a direction parallel to the width direction of the upper swing body 3. The antennas 21 and 22 receive the GNSS radio wave from the positioning satellite 200 and output a signal corresponding to the received GNSS radio wave. The antennas 21 and 22 may be antennas for GPS (Global Positioning System). In the following description, the antennas 21 and 22 are appropriately referred to as GNSS antennas 21 and 22. The position calculation device 23 detects the position of the hydraulic excavator 1 by using, for example, GNSS.

The image pickup device 19 images the work target WA shown in FIG. 1, and the distance detection device 20 obtains the distance from itself (a predetermined position of the hydraulic excavator 1) to the work target WA, so that the work target WA is as wide as possible. It is preferable to obtain information from. Therefore, in the embodiment, the antenna 24, the image pickup device 19, and the distance detection device 20 are installed above the driver's cab 4 of the upper swing body 3. The place where the image pickup device 19 and the distance detection device 20 are installed is not limited to the upper part of the driver's cab 4. For example, the image pickup device 19 and the distance detection device 20 may be installed inside and above the driver's cab 4.

In the image pickup apparatus 19, the incident surface 19L faces the front of the upper swing body 3. In the distance detection device 20, the incident surface 20L faces the front of the upper swing body 3. In an embodiment, the imaging device 19 is a monocular camera including an image sensor such as a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor. In an embodiment, the distance detection device 20 is a three-dimensional laser range finder or a distance sensor. The image pickup device 19 and the distance detection device 20 are not limited thereto. For example, instead of the image pickup device 19 and the distance detection device 20, a device having both a function of acquiring an image of the work target WA and a function of obtaining the distance to the work target WA may be used. Examples of such a device include a stereo camera.

<Control system of hydraulic excavator 1>
FIG. 2 is a diagram showing a control system 1S of a hydraulic excavator 1 which is a work machine according to an embodiment. The control system 1S includes a communication device 25, a sensor controller 26, a work equipment control device 27, an image pickup device 19, a distance detection device 20, a position calculation device 23, an attitude detection device 32, and an IMU (Inertial Measurement Unit). : Inertial measurement unit) 33 and a hydraulic system 36. The communication device 25, the sensor controller 26, and the work equipment control device 27 are connected by a signal line 35. With such a structure, the communication device 25, the sensor controller 26, and the work equipment control device 27 can exchange information with each other via the signal line 35. As the signal line for transmitting information in the control system 1S, an in-vehicle signal line such as CAN (Controller Area Network) is exemplified.

The sensor controller 26 has a processor such as a CPU and a storage device such as a RAM and a ROM. The detection value of the position calculation device 23, the information of the image captured by the image pickup device 19, the detection value of the distance detection device 20, the detection value of the posture detection device 32, and the detection value of the IMU 33 are input to the sensor controller 26. The sensor controller 26 transmits the input detection value and image information to the processing device 51 of the facility 50 shown in FIG. 1 via the signal line 35 and the communication device 25.

The work equipment control device 27 has a processor such as a CPU and a storage device such as a RAM and a ROM. The work machine control device 27 acquires a command for operating at least one of the work machine 2 and the upper swing body 3 generated by the processing device 51 of the facility 50 via the communication device 25. The work equipment control device 27 controls the electromagnetic control valve 28 of the hydraulic system 36 based on the acquired command.

The hydraulic system 36 includes an electromagnetic control valve 28, a hydraulic pump 29, and a hydraulic actuator such as a boom cylinder 10, an arm cylinder 11, a bucket cylinder 12, and a swivel motor 30. The hydraulic pump 29 is driven by the engine 31 to discharge hydraulic oil for operating the hydraulic actuator. The work equipment control device 27 controls the flow rate of the hydraulic oil supplied to the boom cylinder 10, the arm cylinder 11, the bucket cylinder 12, and the swivel motor 30 by controlling the electromagnetic control valve 28. In this way, the work equipment control device 27 controls the operations of the boom cylinder 10, the arm cylinder 11, the bucket cylinder 12, and the swivel motor 30.

The sensor controller 26 acquires the detected values of the first stroke sensor 16, the second stroke sensor 17, and the third stroke sensor 18. The first stroke sensor 16 is provided in the boom cylinder 10, the second stroke sensor 17 is provided in the arm cylinder 11, and the third stroke sensor 18 is provided in the bucket cylinder 12.

The first stroke sensor 16 detects the boom cylinder length, which is the length of the boom cylinder 10, and outputs the boom cylinder length to the sensor controller 26. The second stroke sensor 17 detects the arm cylinder length, which is the length of the arm cylinder 11, and outputs it to the sensor controller 26. The third stroke sensor 18 detects the bucket cylinder length, which is the length of the bucket cylinder 12, and outputs the bucket cylinder length to the sensor controller 26.

Once the boom cylinder length, arm cylinder length, and bucket cylinder length are determined, the posture of the work machine 2 is determined. Therefore, the first stroke sensor 16, the second stroke sensor 17, and the third stroke sensor 18 that detect them correspond to the posture detection device 32 that detects the posture of the work equipment 2. The posture detection device 32 is not limited to the first stroke sensor 16, the second stroke sensor 17, and the third stroke sensor 18, and may be an angle detector.

The sensor controller 26 calculates the inclination angle of the boom 6 with respect to the direction orthogonal to the horizontal plane in the vehicle body coordinate system, which is the coordinate system of the hydraulic excavator 1, from the boom cylinder length detected by the first stroke sensor 16. The sensor controller 26 calculates the inclination angle of the arm 7 with respect to the boom 6 from the arm cylinder length detected by the second stroke sensor 17. The sensor controller 26 calculates the inclination angle of the bucket 8 with respect to the arm 7 from the bucket cylinder length detected by the third stroke sensor 18. The inclination angles of the boom 6, the arm 7, and the bucket 8 are information indicating the posture of the work machine 2. That is, the sensor controller 26 requests information indicating the posture of the work machine 2. The sensor controller 26 transmits the calculated tilt angle to the processing device 51 of the facility 50 shown in FIG. 1 via the signal line 35 and the communication device 25.

The position calculation device 23 includes a processor such as a CPU and a storage device such as a RAM and a ROM. The position calculation device 23 obtains the position of the hydraulic excavator 1. Specifically, the position calculation device 23 detects and outputs the position of the antennas 21 and 22 and the orientation of the upper swing body 3 in the global coordinate system by using the signals acquired from the antennas 21 and 22. The orientation of the upper swivel body 3 represents the orientation of the upper swivel body 3 in the global coordinate system. The orientation of the upper swivel body 3 can be represented by, for example, the orientation of the upper swivel body 3 in the front-rear direction around the vertical axis of the global coordinate system. The azimuth is the rotation angle of the reference axis in the front-rear direction of the upper swivel body 3 around the vertical axis of the global coordinate system. The azimuth represents the orientation of the upper swivel body 3.

The IMU 33 detects the operation and posture of the hydraulic excavator 1. The operation of the hydraulic excavator 1 includes at least one of the operation of the upper swing body 3 and the operation of the traveling device 5. The posture of the hydraulic excavator 1 is represented by the roll angle, pitch angle, and yaw angle of the hydraulic excavator 1. In the embodiment, the IMU 33 detects and outputs the angle or angular velocity and acceleration of the hydraulic excavator 1. The angles of the hydraulic excavator 1 detected by the IMU 33 are the roll angle, pitch angle, and yaw angle of the hydraulic excavator 1.

<About the coordinate system>
FIG. 3 is a diagram for explaining the coordinate system of the image display system 100 and the remote control system 101 according to the embodiment. FIG. 4 is a rear view of the hydraulic excavator 1. FIG. 5 is a diagram illustrating a coordinate system of the image pickup apparatus 19 and the distance detection apparatus 20.

In the image display system 100 and the remote control system 101, there are a global coordinate system, a vehicle body coordinate system, a coordinate system of the image pickup device 19, and a coordinate system of the distance detection device 20. The global coordinate system (Xg, Yg, Zg) is a three-dimensional coordinate system with reference to the origin PG fixed to the earth. In embodiments, the global coordinate system is, for example, a coordinate system in GNSS or GPS.

The vehicle body coordinate system is a three-dimensional coordinate system represented by (Xm, Ym, Zm) with reference to the hydraulic excavator 1. In the embodiment, the origin position PL of the vehicle body coordinate system is the intersection of the Zm axis, which is the rotation center axis of the upper swivel body 3, and the plane orthogonal to the Zm axis in the swing circle of the upper swivel body 3. It is not limited. The Xm axis is an axis extending in the front-rear direction of the upper swing body 3 and orthogonal to the Zm axis. The Xm axis is a reference axis in the front-rear direction of the upper swing body 3. The Ym axis is an axis extending in the width direction of the upper swing body 3 orthogonal to the Zm axis and the Xm axis. The plane orthogonal to the Zm axis in the swing circle can be a plane passing through the center of the swing circle in the Zm axis direction.

In the embodiment, the coordinate system of the image pickup device 19 (hereinafter, appropriately referred to as the image pickup device coordinate system) has the center of the image pickup surface 19P of the image pickup device 19RC as the origin PC as shown in FIG. 5 (Xc). , Yc, Zc) is a three-dimensional coordinate system. The Xc axis of the image pickup device coordinate system (Xc, Yc, Zc) is an axis that passes through the optical center of the image pickup device 19 and extends in a direction orthogonal to the image pickup surface 19P. The Yc axis is an axis orthogonal to the Xc axis. The Zc axis is an axis orthogonal to both the Xc axis and the Yc axis.

In the embodiment, as shown in FIG. 5, the coordinate system of the distance detection device 20 has three-dimensional coordinates represented by (Xd, Yd, Zd) with the center of the detection surface 20P of the distance detection element 20RC as the origin PD. It is a system.

<Attitude of hydraulic excavator 1>
As shown in FIG. 4, the inclination angle θ4 with respect to the left-right direction of the upper swing body 3, that is, the width direction is the roll angle of the hydraulic excavator 1, and the inclination angle θ5 with respect to the front-rear direction of the upper swing body 3 is the pitch of the hydraulic excavator 1. It is an angle, and the angle of the upper swing body 3 around the vertical axis is the yaw angle of the hydraulic excavator 1. The roll angle, pitch angle and yaw angle are obtained by integrating the angular velocities detected by the IMU 33 over time.

The acceleration and angular velocity detected by the IMU 33 are output to the sensor controller 26 as operation information. The sensor controller 26 performs processing such as filtering and integration on the operation information acquired from the IMU 33 to obtain an inclination angle θ4 which is a roll angle, an inclination angle θ5 which is a pitch angle, and a yaw angle. The sensor controller 26 shows the obtained inclination angle θ4, inclination angle θ5, and yaw angle as information related to the posture of the hydraulic excavator 1 via the signal line 35 and the communication device 25 shown in FIG. It is transmitted to the processing device 51 of the facility 50.

As described above, the sensor controller 26 requests information indicating the posture of the work machine 2. Specifically, the information indicating the posture of the work equipment 2 is the inclination angle θ1 of the boom 6 with respect to the direction orthogonal to the horizontal plane (Zm axis direction) in the vehicle body coordinate system, the inclination angle θ2 of the arm 7 with respect to the boom 6, and the arm 7 with respect to the arm 7. The inclination angle θ3 of the bucket 8. The processing device 51 of the facility 50 shown in FIG. 1 has information indicating the posture of the working machine 2 acquired from the sensor controller 26 of the hydraulic excavator 1, that is, the positions of the cutting edge 8T of the bucket 8 from the inclination angles θ1, θ2, θ3 (hereinafter, In the above, P4 (referred to as the cutting edge position) is calculated as appropriate.

The storage unit 51M of the processing device 51 stores the data of the work machine 2 (hereinafter, appropriately referred to as the work machine data). The work equipment data includes the length L1 of the boom 6, the length L2 of the arm 7, and the length L3 of the bucket 8. As shown in FIG. 3, the length L1 of the boom 6 corresponds to the length from the boom pin 13 to the arm pin 14. The length L2 of the arm 7 corresponds to the length from the arm pin 14 to the bucket pin 15. The length L3 of the bucket 8 corresponds to the length from the bucket pin 15 to the cutting edge 8T of the bucket 8. The blade tip 8T is the tip of the blade 8B shown in FIG. The work equipment data includes information on the position up to the boom pin 13 with respect to the origin position PL of the vehicle body coordinate system. The processing device 51 can obtain the cutting edge position P4 with respect to the origin position PL by using the lengths L1, L2, L3, the inclination angles θ1, θ2, θ3 and the origin position PL. In the embodiment, the processing device 51 of the facility 50 obtains the cutting edge position P4, but the sensor controller 26 of the hydraulic excavator 1 may obtain the cutting edge position P4 and transmit it to the processing device 51 of the facility 50. Since the cutting edge position P4 is the position of the cutting edge 8T of the bucket 8 which is a part of the working machine 2, it is the position of the working machine 2. In the former case, the posture detection device 32, the sensor controller 26, and the processing device 51 correspond to the position detection unit that detects the position of the work machine. In the latter case, the posture detection device 32 and the sensor controller 26 correspond to a position detection unit that detects the position of the work machine.

The position detection unit is not limited to the attitude detection device 32, the sensor controller 26 and the processing device 51, or the attitude detection device 32 and the sensor controller 26 described above. For example, the position of the working machine may be detected by a distance measuring device such as a stereo camera or a laser scanner. In this case, the distance measuring device corresponds to the position detecting unit that detects the position of the working machine.

<Control example executed by the image display system 100 and the remote control system 101>
FIG. 6 is a flowchart of a control example executed by the image display system 100 and the remote control system 101 according to the embodiment. FIG. 7 is a diagram showing the image pickup device 19, the distance detection device 20, and the work target WA.

In step S101, the sensor controller 26 shown in FIG. 2 acquires the information of the hydraulic excavator 1. The information of the hydraulic excavator 1 is information obtained from the image pickup device 19, the distance detection device 20, the position calculation device 23, the posture detection device 32, and the IMU 33. As shown in FIG. 7, the image pickup apparatus 19 images the work target WA within the image pickup range TA, and obtains an image of the work target WA. The distance detection device 20 detects the distance Ld from the distance detection device 20 to the work target WA and other objects existing in the detection range MA. The position calculation device 23 obtains reference position information Pga1 and Pga2 corresponding to the positions P1 and P2 of the GNSS antennas 21 and 22 in the global coordinate system. The attitude detection device 32 detects the boom cylinder length, the arm cylinder length, and the bucket cylinder length. The IMU 33 detects the posture of the hydraulic excavator 1, more specifically, the roll angle θ4, the pitch angle θ5, and the yaw angle of the upper swing body 3.

In step S102, the processing device 51 of the image display system 100 and the remote operation system 101 is flood-controlled from the sensor controller 26 of the hydraulic excavator 1 via the communication device 25 of the hydraulic excavator 1 and the communication device 54 connected to the processing device 51. Acquire the information of the excavator 1.

The information of the hydraulic excavator 1 acquired by the processing device 51 from the sensor controller 26 includes an image of the work target WA captured by the image pickup device 19 and the distance detection device 20 to the work target WA detected by the distance detection device 20. The distance information, the posture information of the work machine 2 included in the hydraulic excavator 1 detected by the posture detecting device 32, the reference position information Pga1 and Pga2, and the posture information of the hydraulic excavator 1 are included.

The information on the distance from the distance detection device 20 to the work target WA includes the distance Ld to the work target WA or the object OB existing in the detection range MA and the information on the orientation of the position Pd corresponding to the distance Ld. In the example shown in FIG. 7, the distance Ld is shown as the distance to the work target WA. The information on the orientation of the position Pd is the orientation of the position Pd with respect to the distance detection device 20, and is an angle with respect to each axis Xd, Yd, Zd of the coordinate system of the distance detection device 20. The posture information of the work machine 2 acquired by the processing device 51 is the inclination angles θ1, θ2, and θ3 of the work machine 2 obtained by the sensor controller 26 using the boom cylinder length, the arm cylinder length, and the bucket cylinder length. The posture information of the hydraulic excavator 1 is the roll angle θ4, the pitch angle θ5, and the yaw angle of the hydraulic excavator 1, more specifically, the upper swing body 3.

The processing device 51 includes the inclination angles θ1, θ2, and θ3 of the work machine 2 acquired from the sensor controller 26, the length L1 of the boom 6 stored in the storage unit 51M, the length L2 of the arm 7, and the length of the bucket 8. The cutting edge position P4 of the bucket 8 is obtained by using the L3. The cutting edge position P4 of the bucket 8 is a set of coordinates in the vehicle body coordinate system (Xm, Ym, Zm) of the hydraulic excavator 1.

In step S103, the processing device 51 converts the distance Ld to the work target WA into position information using the information on the distance to the work target WA. The position information is the coordinates of the position Pd in the coordinate system (Xd, Yd, Zd) of the distance detection device 20. In step S103, all distances Ld detected by the distance detection device 20 within the detection range MA are converted into position information. The processing device 51 converts the distance Ld into position information by using the distance Ld and the directional information of the position Pd corresponding to the distance Ld. In step S103, the distance to the object OB existing in the detection range MA is also converted into position information in the same manner as the distance Ld of the work target WA. By the process of step S103, information on the position of the work target WA within the detection range MA can be obtained. Information on the shape of the work target WA can be obtained from the information on the position of the work target WA.

The position information and shape information of the work target WA are a set of coordinates of the position Pd in the coordinate system (Xd, Yd, Zd) of the distance detection device 20. The processing device 51 converts the shape information of the work target WA into the value of the image pickup device coordinate system (Xc, Yc, Zc) and then converts it into the value of the vehicle body coordinate system (Xm, Ym, Zm). Next, in step S104, the processing device 51 obtains the occupied area SA.

FIG. 8 is a diagram for explaining the occupied area SA. The occupied area SA is an area occupied by the work machine 2 in the information on the shape of the work target WA. In the example shown in FIG. 8, the bucket 8 of the work machine 2 is within the detection range MA of the distance detection device 20 and is between the distance detection device 20 and the work target WA. Therefore, in the portion of the occupied area SA, the distance to the bucket 8 is detected by the distance detecting device 20 instead of the distance to the work target WA. In step S105, the processing device 51 removes the portion of the occupied area SA from the shape information of the work target WA obtained in step S103.

In removing the portion of the occupied area SA, the processing device 51 stores at least one of the information of the position and the posture detected by the distance detection device 20 according to at least one of the position and the posture of the bucket 8 in, for example, the storage unit 51M. Let me do it. Such information is included in the posture of the work machine 2 of the hydraulic excavator 1 in the present embodiment. The posture of the work machine 2 uses the inclination angles θ1, θ2, θ3 of the work machine 2, the length L1 of the boom 6, the length L2 of the arm 7, and the length L3 of the bucket 8, and the hydraulic excavator 1 is used as necessary. It can be calculated using the posture. Then, the processing device 51 compares the information detected by the distance detecting device 20 with the information stored in the storage unit 51M, and if both match, it can be assumed that the bucket 8 has been detected. Since the portion where both are matched corresponds to the occupied area SA, the processing device 51 removes the portion of the occupied area SA from the information on the shape of the work target WA.

By the process of removing the occupied area SA using the posture of the work machine 2, the processing device 51 generates the reference image 65 as an index showing the position of the work target WA shown in FIG. 1, when the occupied area SA is generated. Since the information of the bucket 8 of the above is not used, the reference image 65 can be accurately generated.

The process of removing the portion of the occupied area SA using the posture of the working machine 2 may be performed by the following method. The information regarding at least one of the position and the posture of the bucket 8 in the vehicle body coordinate system included in the posture of the work machine 2 includes the inclination angles θ1, θ2, θ3 of the work machine 2, the length L1 of the boom 6, and the length of the arm 7. It is obtained from the length L3 of L2 and the bucket 8. In step S103, information on the shape of the work target WA in the vehicle body coordinate system is obtained. In step S105, the processing device 51 removes the position of the bucket 8 from the shape of the work target WA as the occupied area SA by projecting the position of the bucket 8 onto the information on the shape of the work target WA.

FIG. 9 is a diagram showing information on the shape of the work target WA from which the occupied area has been removed. The information IMWA of the shape of the work target WA is a set of coordinates Pmd (Xmd, Ymd, Zmd) in the vehicle body coordinate system (Xm, Ym, Zm). Coordinate information does not exist in the occupied area SA due to the process of step S105. Next, the process proceeds to step S106, and the processing device 51 generates the first image. The first image is an image on the work target WA that includes a part corresponding to the bucket 8 or a part corresponding to a part of the bucket 8 and extends along the direction in which the upper swing body 3 of the hydraulic excavator 1 turns. ..

<First image>
FIG. 10 is a diagram showing an example of the first image IMC. 11 and 12 are diagrams for explaining a process in which the processing device 51 generates the first image IMC. In the embodiment, the first image IMC shown in FIG. 10 shows the position of the cutting edge 8T of the bucket 8 on the work target WA, and the upper swivel body 3 swivels while the work machine 2 maintains the current posture. The position of the cutting edge 8T on the surface of the work target WA in the case is shown. Next, a process in which the processing device 51 generates the first image IMC will be described.

As shown in FIGS. 10 and 12, the first image IMC has a radius of the distance Rbt between the Zm axis and the bucket 8 when viewed from the Zm axis of the vehicle body coordinate system, which is the turning center axis of the upper turning body 3. Moreover, it is an image of an arc centered on a position corresponding to the Zm axis. In the embodiment, the position of the bucket 8 used when determining the distance Rbt is the position of the cutting edge 8T of the bucket 8 existing in the center of the width direction Wb of the bucket 8 as shown in FIG. The position of the bucket 8 used when determining the distance Rbt is not limited to the cutting edge 8T at the center of the width direction Wb of the bucket 8. The distance Rbt is the Zm axis of the vehicle body coordinate system along the direction parallel to the Xm-Ym plane of the vehicle body coordinate system, and the position Pbt (Xmbt, Ymbt, Zmbt) of the cutting edge 8T at the center of the width direction Wb of the bucket 8. Is the distance. The position Pbt (Xmbt, Ymbt, Zmbt) is a position in the vehicle body coordinate system.

The line LC in FIG. 12 shows the center of the bucket 8 in the width direction Wb. The width direction Wb of the bucket 8 is a direction parallel to the direction in which the bucket pin 15 connecting the bucket 8 and the arm 7 extends, and is a direction parallel to the Ym axis of the vehicle body coordinate system. The distance Rbt is the distance between the Zm axis and the cutting edge 8T of the bucket 8 along the direction parallel to the Xm axis in the vehicle body coordinate system.

The position indicated by the first image IMC is the position of the surface WAS of the work target WA. The processing device 51 projects an arc having a radius of the distance Rbt centered on the Zm axis of the vehicle body coordinate system onto the surface WAS of the axis work target WA in a direction parallel to the Zm axis of the vehicle body coordinate system (hereinafter, the present invention). The position Pt (Xmt, Ymt, Zmt) of the intersection) is obtained as appropriate. The image of the intersecting portion becomes the first image IMC.

The position Pt (Xmt, Ymt, Zmt) of the intersecting portion is, for example, the position of the portion where the curved surface having the distance Rbt centered on the Zm axis of the vehicle body coordinate system as the radius and the surface WAS of the work target WA intersect. .. The portion where the straight line VL that passes through the cutting edge 8T of the bucket 8 existing in the center of the width direction Wb of the bucket 8 and extends in the direction parallel to the Zm axis of the vehicle body coordinate system intersects the surface WAS of the work target WA is the intersection portion. It is a part of the position. The processing device 51 generates the first image IMC by using the information of the position Pt (Xmt, Ymt, Zmt) of the intersecting portion. The first image IMC corresponds to the information of the position Pt (Xmt, Ymt, Zmt) of the intersection.

The first position Ptc in the position Pt (Xmt, Ymt, Zmt) of the intersection is the position of the bucket 8 on the surface WAS of the work target WA facing the bucket 8, and in the embodiment, the center of the width direction Wb of the bucket 8. Corresponds to the position of the cutting edge 8T of the bucket 8 existing in. The first position Ptc is not limited to the position of the cutting edge 8T in the center of the width direction Wb, and may be, for example, a position at one end of the width direction Wp.

The position of the portion extending from the first position Ptc along the turning direction Rtd of the upper swivel body 3 is the portion of the intersecting portion positions Pt (Xmt, Ymt, Zmt) extending from the first position Ptc along the turning direction Rtd. The position of. The first image IMC corresponds to the information of the first position Ptc and the information of the position of the portion extending from the first position Ptc along the turning direction Rtd. That is, the first image IMC is generated based on the information of the first position Ptc and the position information of the portion extending from the first position Ptc along the turning direction Rtd.

In the embodiment, the processing device 51 also causes the display device 52 shown in FIG. 1 to display the first straight line image 62 which is an image of the straight line LV1 and the second straight line image 63 which is an image of the straight line LV2. The straight line LV1 is a straight line extending from the position Pb1 outside the blade 8B on one end 8Wt1 side of the width direction Wb of the bucket 8 to the surface WAS of the work target WA along the direction parallel to the Zm axis of the vehicle body coordinate system. Is. The straight line LV2 is a straight line extending from the position Pb2 outside the blade 8B on the other end 8Wt2 side of the width direction Wb of the bucket 8 to the surface WAS of the work target WA along the direction parallel to the Zm axis of the vehicle body coordinate system. Is. The position of the portion where the straight line LV1 and the surface WAS of the work target WA intersect is Pt1, and the position of the portion where the straight line LV2 and the surface WAS of the work target WA intersect is Pt2. Since the position of each part of the bucket 8 in the vehicle body coordinate system and the position of the surface WAS of the work target WA in the vehicle body coordinate system are known, the processing device 51 can obtain the positions Pb1, Pb2, Pt1, Pt2 from these positions. can. If the positions Pb1, Pb2, Pt1, and Pt2 are obtained, the straight line LV1 and the straight line LV2 are also obtained, so that the processing device 51 can generate the first straight line image 62 and the second straight line image 63.

When the first image IMC, the first straight line image 62, and the second straight line image 63 are obtained, the processing device 51 converts them into an image of the viewpoint of the image pickup device 19. The image of the viewpoint of the image pickup device 19 is an image based on the image pickup device 19. Next, a process for converting the first image IMC, the first straight line image 62, and the second straight line image 63 into an image of the viewpoint of the image pickup apparatus 19 will be described.

FIG. 13 is a diagram showing the positional relationship between the origin PC of the image pickup apparatus 19 and the first image IMC, the first straight line image 62, and the second straight line image 63. The image of the viewpoint of the image pickup device 19 is an image when the first image IMC, the first straight line image 62, and the second straight line image 63 are viewed from the origin PC of the image pickup device in the vehicle body coordinate system (Xm, Ym, Zm). ..

The first image IMC, the first straight line image 62, and the second straight line image 63 are images in a three-dimensional space, but the viewpoint image of the image pickup apparatus 19 is a two-dimensional image. Therefore, the processing device 51 puts the first image IMC, the first straight line image 62, and the second straight line image 63 defined in the three-dimensional space, that is, the vehicle body coordinate system (Xm, Ym, Zm) on the two-dimensional plane. Performs a perspective projection transformation to project. The first image IMC, the first straight line image 62, and the second straight line image 63 converted into the image of the viewpoint of the image pickup apparatus 19 are appropriately referred to as a guide image 60 in the following.

FIG. 14 is a diagram showing the reference image 65 and the first image IMC shown in FIG. When the guide image 60 is generated, the process proceeds to step S107, and the processing device 51 generates the reference image 65. The reference image 65 is a line image along the surface WAS of the work target WA. The processing device 51 generates the reference image 65 by using the information on the position of the work target WA, specifically, the information on the position of the work target WA on the surface WAS.

The reference image 65 includes a plurality of first line image RLs radially extending from a position corresponding to the Zm axis, which is the rotation center axis of the upper swivel body 3, and a swivel direction Rtd of the upper swivel body 3 centered on the Zm axis. It includes a plurality of second line image CLs extending along the line. In the embodiment, the plurality of first line image RLs extend radially from the Zm axis when viewed from the Zm axis direction of the vehicle body coordinate system (Xm, Ym, Zm), and the circumferential direction of the circle centered on the Zm axis. It is a line image arranged along. The plurality of second line image CLs are images of arcs or circles extending along the circumferential direction of the circle centered on the Zm axis when viewed from the Zm axis direction of the vehicle body coordinate system (Xm, Ym, Zm). It is arranged along the radial direction of a circle centered on the Zm axis.

Since the first line image RL and the second line image CL are defined by the vehicle body coordinate system (Xm, Ym, Zm), they include three-dimensional information. In the embodiment, the plurality of first line image RLs are arranged at equal intervals at each angle α along the circumferential direction of the circle centered on Zm. The plurality of second line image CLs are arranged at equal intervals for each distance ΔR along the radial direction of the circle centered on Zm.

After the processing device 51 generates the first line image RL and the second line image CL, the processing device 51 converts them into an image of the viewpoint of the image pickup device 19 to generate a reference image 65. By converting the first line image RL and the second line image CL into the image of the viewpoint of the image pickup device 19, the display device 52 deforms and displays the reference image 65 according to the shape of the work target WA. be able to.

Next, in step S108, the processing device 51 removes the occupied area SA described above from the generated guide image 60 and the reference image 65. Since the guide image 60 and the reference image 65 are generated from the information on the shape of the work target WA, removing the occupied area SA from the guide image 60 and the reference image 65 makes the occupied area SA information on the shape of the work target WA. Means to remove from. The processing device 51 may generate the guide image 60 and the reference image 65 by using the information on the shape of the work target WA from which the occupied area SA is removed.

In step S108, the processing device 51 converts the occupied area SA into an image of the viewpoint of the image pickup device 19 and removes it from the guide image 60 and the reference image 65. The processing device 51 includes a first image IMC, a first straight line image 62, and a second straight line image 63 before being converted into an image of the viewpoint of the image pickup device 19, and a first image before being converted into an image of the viewpoint of the image pickup device 19. The occupied area SA before being converted into the image of the viewpoint of the image pickup apparatus 19 may be removed from the line image RL of 1 and the line image CL of the second line, respectively.

FIG. 15 is a diagram showing a working image 69. In step S109, the processing device 51 synthesizes the guide image 60 from which the occupied area SA has been removed, the reference image 65, and the second image 68 of the work target WA imaged by the image pickup device 19, and is used for work. Image 69 is generated. In step S110, the processing device 51 causes the display device 52 to display the generated working image 69. The work image 69 is an image in which the image 68 of the work target WA, the reference image 65, and the guide image 60 are combined.

Since the reference image 65 is a plurality of line images extending to the surface WAS of the work target WA along the circumferential direction and the radial direction of the circle centered on the turning center axis of the upper swing body 3, the operator of the hydraulic excavator 1 can refer to the reference image 65. By referring to the image 65, the position of the work target WA can be grasped. For example, the operator can grasp the depth, that is, the position of the upper swivel body 3 included in the hydraulic excavator 1 in the front-rear direction by the second line image CL, and the position of the upper swivel body 3 in the swivel direction Rtd by the first line image RL. Can be grasped.

Since the distance Rbt changes according to the distance between the bucket 8 and the Zm axis, the first image IMC also moves away from the hydraulic excavator 1 or approaches the hydraulic excavator 1 according to the distance between the bucket 8 and the Zm axis. Since the first image IMC moves according to the current position of the bucket 8, the operator can grasp the positions of the work target WA and the bucket 8 by operating the work machine 2 while referring to the first image IMC. can. As a result, a decrease in work efficiency is suppressed. In addition, a decrease in work accuracy is suppressed.

By referring to the first image IMC and the second line image CL, the operator refers to the position of the excavation target farther than the bucket 8 when the upper swivel body 3 turns in the current posture of the working machine 2. It is easy to know whether it is on the side or near. As a result, a decrease in work efficiency is suppressed. In addition, a decrease in work accuracy is suppressed.

The first straight line image 62 and the second straight line image 63 show the positions of the cutting edge 8T of the bucket 8 on the surface WAS of the work target WA. That is, the portion sandwiched between the first straight line image 62 and the second straight line image 63 on the surface WAS of the work target WA is the position of the cutting edge 8T of the bucket 8. Therefore, the operator can more easily grasp the positional relationship between the bucket 8 and the work target WA from the first straight line image 62 and the second straight line image 63. As a result, a decrease in work efficiency is suppressed. In addition, a decrease in work accuracy is suppressed.

Since the reference image 65 and the first image IMC are displayed along the shape of the terrain of the target to be worked by the work target WA, for example, the hydraulic excavator 1, the reference image 65 and the first image IMC are displayed on the terrain surface displayed in two dimensions on the display device 52. The relative positional relationship between the image 65 and the first image IMC can be easily grasped. Further, since the first line image RL and the second line image CL constituting the reference image 65 are arranged at equal intervals in the vehicle body coordinate system, the operator can easily grasp the sense of distance on the terrain surface. It becomes easier to grasp the perspective.

In the embodiment, the working image 69 may include information 64 indicating the distance between the cutting edge 8T of the bucket 8 and the work target WA. The information 64 has an advantage that the operator can grasp the actual distance between the cutting edge 8T of the bucket 8 and the work target WA. The distance between the cutting edge 8T of the bucket 8 and the work target WA can be the distance from the cutting edge 8T at the center of the width direction Wb of the bucket 8 to the surface WAS of the work target WA.

The information 64 may be spatial position information regarding the work tool or the work target WA. The spatial position information is replaced with the distance between the cutting edge 8T of the bucket 8 and the work target WA, or in addition to the distance, information on the posture such as the angle of the bucket 8, information indicating the relative distance between the bucket 8 and the work target WA, Information indicating the relationship between, for example, the orientation of the cutting edge 8T of the bucket 8 and the orientation of the surface of the work target WA, information indicating the position of the bucket 8 in coordinates, information indicating the orientation of the surface of the work target WA, and the bucket from the imaging device 19. It includes information such as information indicating the distance in the Xm direction in the vehicle body coordinate system up to the cutting edge 8T of 8.

That is, the processing device 51 includes the position of the bucket 8 which is a work tool, the posture of the bucket 8, the position of the work target WA, the relative posture of the work target WA, the relative distance between the bucket 8 and the work target WA, and the bucket. At least one of the relative postures between the 8 and the work target WA may be obtained and displayed on the display device 52.

<Reference image related to the modified example>
FIG. 16 is a diagram showing a reference image 65a according to a modified example. The reference image 65a is different from the reference image 65 in which the vehicle body coordinate system is the polar coordinate system, and the plurality of first linear images 66 and these are viewed from the direction of the Zm axis in the vehicle body coordinate system (Xm, Ym, Zm). It is an image of a lattice formed by a plurality of second linear images 67 orthogonal to. The plurality of first linear images 66 are parallel to the Xm axis of the vehicle body coordinate system, and the plurality of second linear images 67 are parallel to the Ym axis of the vehicle body coordinate system. The intervals at which the plurality of first linear images 66 are arranged are equal. Further, the intervals at which the plurality of second linear images 67 are arranged are equal. Then, the interval at which the plurality of first linear images 66 are arranged is equal to the interval at which the plurality of second linear images 67 are arranged.

The processing device 51 synthesizes the guide image 60 from which the occupied area SA has been removed, the reference image 65a, and the second image 68 of the work target WA imaged by the image pickup device 19 to generate a work image 69. do. Even when the processing device 51 displays such a grid image on the display device 52 together with the guide image 60, the operator can easily grasp the positional relationship between the position of the bucket 8 and the work target WA during turning and excavation. Can be done.

<Third image>
FIG. 17 is a diagram showing an example of the first image IMC and the third image 61. The third image 61 is a line image connecting the position Pt1 and the position Pt2 and along the surface WAS of the work target WA. The position Pt1 is the position of the portion where the straight line LV1 corresponding to the first straight line image 62 and the surface WAS of the work target WA intersect. The position Pt2 is the position of the portion where the straight line LV2 corresponding to the second straight line image 63 and the surface WAS of the work target WA intersect. The processing device 51 uses a third image 61 as a set of positions of the surface WAS when a straight line connecting the first position Pt1 and the second position Pt2 is projected onto the surface WAS of the work target WA. The position Pt where the straight line parallel to the Zm axis of the vehicle body coordinate system intersects the surface WAS of the work target WA, which extends from the cutting edge 8T at the center position in the width direction Wb of the bucket 8, is a part of the third image 61. .. The position Pt is also part of the first image IMC.

The processing device 51 may display the third image 61 on the display device 52 in addition to the first image IMC. The third image 61 allows the operator to grasp the current position of the bucket 8 on the surface WAS of the work target WA. As a result, the operator can easily and surely grasp the positional relationship between the bucket 8 and the working position. When the processing device 51 displays the first image IMC and the third image 61 on the display device 52, the display modes of the two may be different. To make the display form different, for example, the color of the first image IMC and the color of the third image 61 are different, and the line thickness of the first image IMC and the line thickness of the third image 61 are different. It includes making the line type of the first image IMC different from the line type of the third image 61, and combining two or more of them. By making the display form of the first image IMC and the third image 61 different, the operator can easily distinguish the third image 61 from the first image IMC.

<Control system of hydraulic excavator 1 according to the modified example>
FIG. 18 is a diagram showing a control system 1Sa of the hydraulic excavator 1 according to the modified example. The image display system 100 and the remote control system 101 described above remotely controlled the hydraulic excavator 1 by using the operation device 53 of the facility 50 shown in FIG. In this modification, the display device 52a is provided in the driver's cab 4 shown in FIG. In this modification, in order for the hydraulic excavator 1 to assist the operator's work, a work image 69 is displayed on the display device 52a.

In the control system 1Sa, the processing device 51 and the operation device 53a are connected to the signal line 35 of the control system 1S described above. A display device 52a is connected to the processing device 51. The processing device 51 included in the control system 1Sa has the same functions as the processing device 51 provided in the facility 50 shown in FIG. 1 in the image display system 100 and the remote control system 101 described above. The processing device 51 and the display device 52a constitute an image display system 100a according to a modified example.

In the modified example, the processing device 51 may obtain the cutting edge position P4 using the detection values of the posture detection device 32 and the sensor controller 26, or the sensor controller 26 may use the detection values of the posture detection device 32 to obtain the cutting edge position P4. You may ask. In the former case, the position detection device 32, the sensor controller 26, and the processing device 51 correspond to the position detection unit that detects the position of the work tool. In the latter case, the position detection device 32 and the sensor controller 26 correspond to a position detection unit that detects the position of the work tool. The position detection unit may be a distance measuring device such as a stereo camera or a laser scanner described above.

The display device 52a of the control system 1Sa is a head-up display that projects an image onto the front window FG of the driver's seat. The display device 52a may be a head-mounted display. The display device 52a displays the work image 69 at a position aligned with the bucket 8 that can be seen through the front window FG. In this case, since the work target WA and the bucket 8 are visible through the front window FG, the work image 69 displayed by the display device 52a may include at least the first image IMC. That is, when the display device 52a is a head-up display or a head-mounted display, the processing device 51 causes the display device 52a to display the first image IMC. In the modified example, in addition to the first image IMC, the reference image 65 is also displayed on the display device 52a.

The display device 52a is not limited to the head-up display, and may be a normal display. When the display device 52a is a normal display, the processing device 51 displays on the display device 52a a working image 69 in which at least the first image IMC and the second image 68 captured by the image pickup device 19 are combined. Let me.

The operation device 53a is a device for operating the hydraulic excavator 1, and includes a left operation lever 53La and a right operation lever 53Ra. The operating device 53a may be of a pilot hydraulic system or an electric system.

The hydraulic excavator 1 including the control system 1Sa includes at least the first image IMC when the work target WA is viewed from a predetermined position, for example, a position corresponding to the viewpoint of the operator seated in the driver's seat by the image display system 100a. The working image 69 is displayed on the front window FG. By such processing, the operator can easily and surely grasp the positional relationship between the bucket 8 and the working position by the first image IMC displayed on the image on the front window FG. As a result, the image display system 100a can suppress a decrease in work efficiency and a decrease in work accuracy. The farther the bucket 8 is from the vehicle body 1B of the hydraulic excavator, the more difficult it is for the operator to grasp the perspective. In such a case, the image display system 100a can suppress a decrease in work efficiency and work accuracy.

Further, even an inexperienced operator can easily grasp the perspective of the bucket 8 by using the hydraulic excavator 1 provided with the image display system 100a. As a result, a decrease in work efficiency is suppressed. In addition, a decrease in work accuracy is suppressed. Further, even in a situation where it is difficult for the operator to visually recognize the actual work target WA such as night work, the operator can work while looking at the work image 69 displayed on the display device 52a. Deterioration of efficiency and work accuracy is suppressed.

The image display systems 100, 100a and the remote control system 101 superimpose the guide image 60 and the reference image 65 or 65a generated from the viewpoint of the image pickup device 19 on the image 68 of the actual work target WA captured by the image pickup device 19. At the same time, it is displayed on the display device 52. By such processing, the image display system 100 and the remote control system 101 give the operator who remotely controls the hydraulic excavator 1 using the image of the work target WA displayed on the display device 52 the position of the bucket 8 and the work target WA. It is possible to make it easier to grasp the positional relationship with. As a result, a decrease in work efficiency is suppressed. In addition, a decrease in work accuracy is suppressed.

Since the first image IMC included in the guide image 60 extends along the turning direction of the upper swivel body 3, the operator can work with the bucket 8 when the upper swivel body 3 turns by referring to the guide image 60. The relative positional relationship with the work point in the target WA can be easily grasped. As a result, the image display system 100 and the remote control system 101 can suppress a decrease in work efficiency and a decrease in work accuracy in a work machine having a swivel body.

Even an inexperienced operator can easily grasp the positional relationship between the position of the bucket 8 and the work point in the work target WA by using the image display systems 100 and 100a and the remote control system 101. As a result, deterioration of work efficiency and work accuracy is suppressed. Further, the image display system 100 and the remote control system 101 superimpose the guide image 60, the reference image 65 or 65a, and the image 68 of the actual work target WA on the display device 52, thereby displaying the image during work. By making the screen that the operator pays attention to as a single screen, it is possible to suppress a decrease in work efficiency.

In the reference image 65 or 65a, the intervals between the adjacent first line images RL are equal, and the intervals between the adjacent second line images CL are equal. Therefore, by superimposing and displaying the reference image 65 or 65a and the image 68 of the actual work target WA captured by the imaging device 19, it becomes easy to grasp the work point on the work target WA. Further, by superimposing the first image IMC of the guide image 60 and the reference image 65, the operator can easily grasp the distance moved by the bucket 8, so that the decrease in work efficiency is suppressed.

Since the occupied area SA, which is the area of the work machine 2, is removed from the guide image 60 and the reference image 65 or 65a, the guide image 60 and the reference image 65 or 65a are distorted by the occupied area SA and the guide image is applied to the work machine 2. It is possible to avoid displaying 60 and the reference image 65 in an overlapping manner. As a result, the image display system 100 and the remote control system 101 can display the work image 69 on the display device 52 in a form that is easy for the operator to see.

In the embodiments and modifications, the guide image 60 may include at least the first image IMC. That is, the guide image 60 does not have to include the first straight line image 62 and the second straight line image 63. Further, the processing device 51 may change the display form of, for example, the first image IMC of the guide image 60 according to the distance between the cutting edge 8T of the bucket 8 and the work target WA. By doing so, the operator can easily grasp the position of the bucket 8 and the distance between the work target WA.

In the embodiment, the processing device 51 may display at least the first image IMC and the second image 68 on the display device 52. Even in this case, the operator refers to the first image IMC extending along the turning direction Rtd of the upper swivel body 3 so that the bucket 8 and the work point in the work target WA when the upper swivel body 3 turns are relative to each other. Positional relationship can be easily grasped. As a result, by displaying at least the first image IMC and the second image 68 on the display device 52, the decrease in work efficiency and the decrease in work accuracy are suppressed in the work machine having the swivel body. ..

In the embodiment, the processing device 51 generates a working image 69 in the vehicle body coordinate system (Xm, Ym, Zm), but the global coordinate system (Xg, Yg, Zg) and the image pickup device coordinate system (Xc, Yc, The working image 69 may be generated in either Zc) or the coordinate system (Xd, Yd, Zd) of the distance detection device 20. When the working image 69 is generated in the global coordinate system (Xg, Yg, Zg), the GNSS antennas 21 and 22 and the position calculation device 23 are required. When the working image 69 is generated in the vehicle body coordinate system (Xm, Ym, Zm), the image pickup device coordinate system (Xc, Yc, Zc) and the distance detection device 20 coordinate system (Xd, Yd, Zd), The GNSS antennas 21 and 22 and the position calculation device 23 are unnecessary.

In the embodiment, the occupied area SA, which is the area of the working machine 2, may not be removed from the guide image 60 and the reference image 65. Even in this case, according to the first image IMC and the second image 68 displayed on the display device 52, the operator can use the relative positional relationship between the bucket 8 and the work point in the work target WA when the upper swivel body 3 is swiveled. Can be easily grasped. Therefore, in a work machine having a swivel body, a decrease in work efficiency is suppressed. Further, in a work machine having a swivel body, the accuracy of work is suppressed.

In the above-described embodiment, a part of the hydraulic excavator 1 detected by the distance detection device 20, for example, the bucket 8 as described above is removed to obtain information on the shape of the work target WA (three-dimensional topographical data). However, the three-dimensional topographical data acquired in the past, for example, a few seconds ago is stored in the storage unit 51M of the processing device 51, and the processing unit 51P of the processing device 51 stores the current work target WA and the stored 3 It is determined whether the dimensional terrain data is at the same position, and if it is at the same position, the reference image 65 or 65a may be displayed using the past three-dimensional terrain data. That is, the processing device 51 can display the reference image 65 or 65a if there is past three-dimensional terrain data even if there is terrain hidden by a part of the hydraulic excavator 1 when viewed from the image pickup device 19. can.

<Example of image display in loading work>
In the embodiment and the modified example, an example of image display by the image display systems 100 and 100a will be described in the loading operation in which the hydraulic excavator 1 loads a load of rocks, earth and sand, etc. into the vessel of the dump truck. The processing device 51 of the image display systems 100 and 100a acquires the position of the dump truck, which is the target of the loading work, detected by the distance detection device 20. Since the first image IMC and the reference image 65 or 65a included in the work image 69 stand up at the position of the dump truck, the operator can grasp the relationship between the work machine 2 and the height and position of the dump truck. By adding the position of the dump truck to the working image 69, the operator can turn the upper swivel body 3 to bring the bucket 8 of the work machine 2 closer to the dump truck, and at what position the bucket 8 is placed on the dump truck. You can figure out if it will reach you. Therefore, the interference between the working machine 2 and the dump truck is suppressed.

The processing device 51 may handle the position of the dump truck detected by the distance detection device 20 as a part of the work target WA, or the position of the dump truck from the management server or the like via the communication device 25 shown in FIG. May be obtained. In the latter case, the processing device 51 may apply the information of the dump truck model to the acquired position of the dump truck and set it as the position of the dump truck.

The processing device 51 compares the height of the dump truck with the height of the working machine 2, specifically the height of the cutting edge 8T of the bucket 8, and even if the display form of the first image IMC is changed according to the comparison result. good. By such processing, the operator can grasp the relationship between the height of the work machine 2 and the height of the dump truck, so that the interference between the work machine 2 and the dump truck is suppressed. When changing the display form of the first image, the processing device 51 displays, for example, the first image IMC in red if the height of the work machine 2 is lower than the height of the dump truck, and the height of the dump truck and the work machine. If the height of the work equipment 2 is equal to the height of the dump truck 2, the first image IMC can be displayed in orange, and if the height of the work equipment 2 is higher than the height of the dump truck, the first image IMC can be displayed in green.

When at least one of the third image 61, the first straight line image 61, and the second straight line image 62 is displayed together with the first image IMC, the processing device 51 together with the first image IMC, depending on the height of the dump track. The display form of at least one of the third image 61, the first straight line image 61, and the second straight line image 62 may be changed. By such processing, the operator can easily grasp the relationship between the height of the work machine 2 and the height of the dump truck.

In the embodiments and modifications, the type of the work machine 2 and the type of the work machine are not limited as long as the work machine includes a swivel body having the work machine 2.

Although the embodiments and modifications have been described above, the embodiments are not limited by the contents described above. In addition, the above-mentioned components include those that can be easily assumed by those skilled in the art, those that are substantially the same, that is, those in a so-called equal range. Furthermore, the components described above can be combined as appropriate. Further, at least one of various omissions, substitutions and changes of the components can be made without departing from the gist of the embodiments and modifications.

1 Hydraulic excavator 1B Vehicle body 1S, 1Sa Control system 2 Working machine 3 Upper swivel body 8 Bucket 8B Blade 8T Cutting edge 19 Imaging device 20 Distance detection device 23 Position calculation device 50 Facility 51 Processing device 51IO Input / output unit 51M Storage unit 51P Processing unit 52, 52a Display device 60 Guide image 61 Third image 62 First straight image 63 Second straight image 65, 65a Reference image 68 Second image 69 Work image 100, 100a Work machine image display system (image display system)
101 Remote control system for work machines (remote control system)
CL 2nd line image RL 1st line image FG Front window IMC 1st image Rtd Turning direction WA Work target WAS Surface Wb Width direction

Claims (8)

It is an image display system of a work machine applied to a work machine having a work tool and a work machine having a swivel body to which the work machine is attached.
A position detection unit that detects the position of the work machine, and
A distance detection device that non-contactly obtains information on the shape of the work object within the detection range,
A first image along the shape of the terrain of the work target is generated by using the position information of the work tool obtained by the position detection unit and the shape information of the work target obtained by the distance detection device. Then, including a processing device to be displayed on the display device,
The first image shows information on a first position corresponding to the work tool on the surface of the work object facing the work tool, and a position of a portion extending from the first position along a direction in which the swivel body turns. Generated along the shape of the terrain of the work target based on the information of
Image display system for work machines. The information on the shape of the work object is three-dimensional topographical data.
The image display system for a work machine according to claim 1. The first image is
It is an image of an arc whose radius is the distance between the turning center axis and the working tool and the position corresponding to the turning center axis is the center when viewed from the direction of the turning center axis of the turning body.
The image display system for the work machine according to claim 1 or 2. The processing device is
A linear image connecting the work tool and the work object is generated and displayed on the display device.
The image display system for a work machine according to any one of claims 1 to 3. The processing device is
A line image along the surface of the work object is generated using the information on the position of the work object, combined with the image of the work object, and displayed on the display device.
The image display system for a work machine according to any one of claims 1 to 4. It has an imaging device attached to the swivel body and has an imaging device.
The processing device is
The first image and the second image, which is an image of the work object captured by the imaging device, are combined and displayed on the display device.
The image display system for a work machine according to any one of claims 1 to 5. The image pickup device, the position detection unit, and the distance detection device are provided in the work machine.
The processing device and the display device are provided in a facility equipped with an operating device for remotely controlling the work machine.
The image display system for a work machine according to claim 6. An image display system of a work machine applied to a work machine having a work tool, a swivel body to which the work machine is attached, a position detection unit for detecting the position of the work machine, a distance detection device, and an image pickup device. And
Display device and
Along with the shape of the terrain of the work target using the position information of the work tool obtained by the position detection unit and the shape information of the work target within the detection range obtained by the distance detection device in a non-contact manner. A processing device that generates a first image, combines it with a second image that is an image of the work object captured by the image pickup device, and displays the image on the display device.
The first image shows information on a first position corresponding to the work tool on the surface of the work object facing the work tool, and a position of a portion extending from the first position along a direction in which the swivel body turns. Generated based on the information in
Image display system for work machines.

Images