WO2019054000A1 - 表示システム、表示方法、及び表示装置 - Google Patents
表示システム、表示方法、及び表示装置 Download PDFInfo
- Publication number
- WO2019054000A1 WO2019054000A1 PCT/JP2018/024065 JP2018024065W WO2019054000A1 WO 2019054000 A1 WO2019054000 A1 WO 2019054000A1 JP 2018024065 W JP2018024065 W JP 2018024065W WO 2019054000 A1 WO2019054000 A1 WO 2019054000A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- image
- work machine
- transport vehicle
- display
- coordinate system
- Prior art date
Links
Images
Classifications
-
- E—FIXED CONSTRUCTIONS
- E02—HYDRAULIC ENGINEERING; FOUNDATIONS; SOIL SHIFTING
- E02F—DREDGING; SOIL-SHIFTING
- E02F9/00—Component parts of dredgers or soil-shifting machines, not restricted to one of the kinds covered by groups E02F3/00 - E02F7/00
- E02F9/26—Indicating devices
- E02F9/261—Surveying the work-site to be treated
-
- E—FIXED CONSTRUCTIONS
- E02—HYDRAULIC ENGINEERING; FOUNDATIONS; SOIL SHIFTING
- E02F—DREDGING; SOIL-SHIFTING
- E02F9/00—Component parts of dredgers or soil-shifting machines, not restricted to one of the kinds covered by groups E02F3/00 - E02F7/00
- E02F9/26—Indicating devices
- E02F9/264—Sensors and their calibration for indicating the position of the work tool
-
- E—FIXED CONSTRUCTIONS
- E02—HYDRAULIC ENGINEERING; FOUNDATIONS; SOIL SHIFTING
- E02F—DREDGING; SOIL-SHIFTING
- E02F9/00—Component parts of dredgers or soil-shifting machines, not restricted to one of the kinds covered by groups E02F3/00 - E02F7/00
- E02F9/20—Drives; Control devices
- E02F9/2025—Particular purposes of control systems not otherwise provided for
-
- E—FIXED CONSTRUCTIONS
- E02—HYDRAULIC ENGINEERING; FOUNDATIONS; SOIL SHIFTING
- E02F—DREDGING; SOIL-SHIFTING
- E02F9/00—Component parts of dredgers or soil-shifting machines, not restricted to one of the kinds covered by groups E02F3/00 - E02F7/00
- E02F9/20—Drives; Control devices
- E02F9/2025—Particular purposes of control systems not otherwise provided for
- E02F9/205—Remotely operated machines, e.g. unmanned vehicles
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
- H04N7/183—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast for receiving images from a single remote source
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60Y—INDEXING SCHEME RELATING TO ASPECTS CROSS-CUTTING VEHICLE TECHNOLOGY
- B60Y2200/00—Type of vehicle
- B60Y2200/40—Special vehicles
- B60Y2200/41—Construction vehicles, e.g. graders, excavators
- B60Y2200/412—Excavators
-
- E—FIXED CONSTRUCTIONS
- E02—HYDRAULIC ENGINEERING; FOUNDATIONS; SOIL SHIFTING
- E02F—DREDGING; SOIL-SHIFTING
- E02F3/00—Dredgers; Soil-shifting machines
- E02F3/04—Dredgers; Soil-shifting machines mechanically-driven
- E02F3/28—Dredgers; Soil-shifting machines mechanically-driven with digging tools mounted on a dipper- or bucket-arm, i.e. there is either one arm or a pair of arms, e.g. dippers, buckets
- E02F3/30—Dredgers; Soil-shifting machines mechanically-driven with digging tools mounted on a dipper- or bucket-arm, i.e. there is either one arm or a pair of arms, e.g. dippers, buckets with a dipper-arm pivoted on a cantilever beam, i.e. boom
- E02F3/32—Dredgers; Soil-shifting machines mechanically-driven with digging tools mounted on a dipper- or bucket-arm, i.e. there is either one arm or a pair of arms, e.g. dippers, buckets with a dipper-arm pivoted on a cantilever beam, i.e. boom working downwardly and towards the machine, e.g. with backhoes
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04Q—SELECTING
- H04Q9/00—Arrangements in telecontrol or telemetry systems for selectively calling a substation from a main station, in which substation desired apparatus is selected for applying a control signal thereto or for obtaining measured values therefrom
Definitions
- the present invention relates to a display system, a display method, and a display device.
- Patent Document 1 discloses a technique for capturing an image of a work site with a camera provided in a work machine, and capturing an image of the work machine from the outside with a camera installed in the work site.
- the work machine When the work machine is remotely operated, if the image displayed on the display device is a two-dimensional image, the operator may have difficulty obtaining a sense of perspective of the work site. If it is difficult to obtain a sense of perspective at the work site, it is difficult for the operator to grasp the distance between the work machine and the transport vehicle when performing loading work on the transport vehicle using the work machine. As a result, the work efficiency of remote control may be reduced.
- An aspect of the present invention is to provide a technology that allows an operator to remotely control a working machine comfortably and to suppress a decrease in work efficiency.
- a virtual viewpoint of the work machine viewed from a virtual viewpoint outside the work machine based on detection data of a position of the work machine having a work machine and detection data of a posture of the work machine A virtual viewpoint image of the transport vehicle viewed from the virtual viewpoint based on a work machine image generation unit that generates a work machine image showing an image, detection data of the position of the transport vehicle, and detection data of the attitude of the transport vehicle
- a transport vehicle image generation unit that generates a transport vehicle image showing the image, a synthesis unit that generates a composite image in which the work machine image and the transport vehicle image are superimposed, and a display that displays the composite image outside the work machine
- a display control unit configured to display on the device.
- the operator can remote-control a working machine comfortably comfortably, and the technique which can suppress the fall of working efficiency is provided.
- FIG. 1 is a view schematically showing an example of a remote control system for a working machine according to the present embodiment.
- FIG. 2 is a side view schematically showing the working machine according to the present embodiment.
- FIG. 3 is a rear view schematically showing the working machine according to the present embodiment.
- FIG. 4 is a plan view schematically showing the working machine according to the present embodiment.
- FIG. 5 is a side view schematically showing the transport vehicle according to the present embodiment.
- FIG. 6 is a rear view schematically showing the transport vehicle according to the present embodiment.
- FIG. 7 is a plan view schematically showing the transport vehicle according to the present embodiment.
- FIG. 8 is a view schematically showing a distance detection device and an imaging device according to the present embodiment.
- FIG. 9 is a view schematically showing an example of the operation of the distance detection device and the imaging device according to the present embodiment.
- FIG. 10 is a functional block diagram showing an example of a control device according to the present embodiment.
- FIG. 11 is a view showing a display example of the display device according to the present embodiment.
- FIG. 12 is a flowchart showing an example of the display method according to the present embodiment.
- FIG. 13 is a flowchart showing an example of the display method according to the present embodiment.
- FIG. 14 is a flowchart showing an example of the display method according to the present embodiment.
- FIG. 15 is a view schematically showing an example of an imposed image according to the present embodiment.
- FIG. 16 is a view schematically showing an example of an imposed image according to the present embodiment.
- FIG. 17 is a functional block diagram showing an example of a control device according to the present embodiment.
- FIG. 1 is a figure which shows typically an example of the remote control system 100 of the working machine 1 which concerns on this embodiment.
- the remote control system 100 remotely controls the work machine 1.
- the working machine 1 is a hydraulic shovel.
- the work machine 1 is appropriately referred to as a hydraulic shovel 1.
- the hydraulic shovel 1 includes a traveling body 2, a swing body 3, a work machine 4, a hydraulic cylinder 5, a distance detection device 20, a position detection device 30 (first position detection device), and a posture detection device 40 1 attitude detection device), an imaging device 50, and a control device 60.
- the hydraulic shovel 1 exists at a work site and works at the work site.
- a vehicle body coordinate system (Xm-Ym-Zm coordinate system) is defined in the revolving unit 3.
- the vehicle body coordinate system is a three-dimensional coordinate system that indicates the relative position with respect to the origin PL defined on the revolving unit 3.
- the vehicle body coordinate system of the revolving unit 3 is defined by the Xm axis of the first reference plane with respect to the origin PL, the Ym axis orthogonal to the Xm axis in the first reference plane, and the Zm axis orthogonal to the first reference plane Be done.
- a direction parallel to the Xm axis is taken as the Xm axis direction
- a direction parallel to the Ym axis is taken as the Ym axis direction
- a direction parallel to the Zm axis is taken as the Zm axis direction.
- the Xm-axis direction is the front-rear direction of the swing body 3
- the Ym-axis direction is the left-right direction of the swing body 3
- the Zm-axis direction is the vertical direction of the swing body 3.
- the traveling body 2 has crawler belts 2A and 2B.
- the hydraulic shovel 1 travels by rotation of the crawler belts 2A and 2B.
- the Zm axis of the vehicle body coordinate system is orthogonal to the ground contact surface of the crawler belts 2A and 2B.
- the + Zm direction of the vehicle body coordinate system is the upper direction of the revolving unit 3 and is the direction away from the ground contact surface of the crawler belts 2A and 2B.
- the -Zm direction of the vehicle body coordinate system is the lower direction of the swing structure 3 and is the direction opposite to the + Zm direction.
- the swing body 3 is swingably supported by the traveling body 2.
- the pivoting body 3 is pivotable about a pivot axis Zr.
- the pivot axis Zr is parallel to the Zm axis of the vehicle coordinate system.
- the origin PL of the vehicle body coordinate system is defined at the center of the swing circle of the swing body 3.
- the center of the swing circle is located on the pivot axis Zr of the swing body 3.
- the work implement 4 is supported by the rotating body 3.
- the vehicle body coordinate system at least a part of the work implement 4 is disposed in the + Xm direction relative to the swing body 3.
- the + Xm direction of the vehicle body coordinate system is the forward direction of the swing body 3 and is the direction in which the tip 9 of the working machine 4 exists with reference to the swing body 3.
- the ⁇ Xm direction of the vehicle body coordinate system is the back direction of the swing structure 3 and is the direction opposite to the + Xm direction.
- the work implement 4 has a boom 6 connected to the swing body 3, an arm 7 connected to the boom 6, and a bucket 8 connected to the arm 7.
- the tip 9 of the work implement 4 includes the cutting edge of the bucket 8.
- the tip 9 of the working machine 4 may be the tip of the blade of the bucket 8.
- the hydraulic cylinder 5 generates power for driving the work implement 4.
- the hydraulic cylinder 5 is driven by hydraulic oil supplied from a hydraulic pump.
- the hydraulic cylinder 5 includes a boom cylinder 10 driving the boom 6, an arm cylinder 11 driving the arm 7, and a bucket cylinder 12 driving the bucket 8.
- the boom 6 is rotatably coupled to the revolving unit 3 around the rotation axis AX1.
- the arm 7 is rotatably coupled to the tip of the boom 6 around the rotation axis AX2.
- the bucket 8 is rotatably coupled to the tip of the arm 7 about the rotation axis AX3.
- the rotation axis AX1 of the boom 6, the rotation axis AX2 of the arm 7, and the rotation axis AX3 of the bucket 8 are parallel to the Ym axis of the vehicle coordinate system.
- the + Ym direction of the vehicle body coordinate system is the left direction of the revolving unit 3.
- the -Ym direction of the vehicle body coordinate system is the direction opposite to the + Ym direction, and is the rightward direction of the revolving unit 3.
- the distance detection device 20 is mounted on the hydraulic shovel 1. The distance detection device 20 detects the distance Ld to the object WAa around the hydraulic shovel 1 (see FIG. 9).
- the target WAa At least one of the construction target to be constructed at the work site, the ground surface, at least a part of the working machine 4 existing in the forward direction with respect to the swing body 3, the structure, and the working machine other than the hydraulic shovel 1 It is illustrated.
- the distance detection device 20 includes a laser range finder.
- the distance detection device 20 includes an emitting unit that emits a laser beam that is detection light, and a light receiving unit that receives the reflected light of the laser beam irradiated to the object WAa.
- the distance detection device 20 is provided on the upper portion of the swing body 3.
- the light receiving unit of the distance detection device 20 faces the front direction (+ Xm direction) of the swing body 3.
- the distance detection device 20 detects the distance Ld between the rotating body 3 and the target WAa in the forward direction. Detection data of the distance detection device 20 is output to the control device 60.
- the position detection device 30 is mounted on the hydraulic shovel 1.
- the position detection device 30 detects the position Pa of the swinging body 3 of the hydraulic shovel 1 in the global coordinate system (Xg-Yg-Zg coordinate system).
- a global coordinate system is a three-dimensional coordinate system based on an origin PG defined on the earth, and is a Global Navigation Satellite System such as the Global Positioning System (GPS). : Coordinate system showing absolute position defined by GNSS.
- GPS Global Positioning System
- the position detection device 30 is provided on the revolving unit 3.
- the position detection device 30 detects the position Pa of the rotating body 3 using the GNSS. Detection data of the position detection device 30 is output to the control device 60.
- the posture detection device 40 is mounted on the hydraulic shovel 1.
- the posture detection device 40 detects the posture of the hydraulic shovel 1.
- the posture of the hydraulic shovel 1 includes the posture of the swing body 3 and the posture of the work implement 4.
- the posture of the swing body 3 includes the inclination angle of the swing body 3.
- the posture of the work implement 4 includes the inclination angle of the work implement 4.
- the attitude detection device 40 includes a swing body attitude sensor 41 that detects the attitude of the swing body 3 and a work machine attitude sensor 42 that detects the attitude of the work machine 4. Detection data of the posture detection device 40 is output to the control device 60.
- the swing body posture sensor 41 includes an inertial measurement unit (IMU) provided in the swing body 3.
- the work implement attitude sensor 42 includes a stroke sensor that detects a cylinder stroke that indicates the amount of operation of the hydraulic cylinder 5.
- the imaging device 50 is mounted on the hydraulic shovel 1.
- the imaging device 50 images a target WAa around the hydraulic shovel 1 to acquire an image of the target WAa.
- the imaging device 50 includes an optical system and an image sensor that receives light passing through the optical system.
- the image sensor includes a CCD (Couple Charged Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor.
- the imaging device 50 is a monocular camera.
- the imaging device 50 is provided on the upper part of the revolving unit 3.
- the incident surface of the optical system of the imaging device 50 faces the front direction (+ Xm direction) of the swing body 3.
- the imaging device 50 acquires an image of the target WAa in the forward direction of the revolving unit 3.
- the imaging device 50 can acquire an image of at least a part of the work implement 4 present in the forward direction relative to the swing body 3.
- the image data of the object WAa acquired by the imaging device 50 is output to the control device 60.
- an image acquired by the imaging device 50 is appropriately referred to as a real image RG.
- the actual image RG shows the actual image of the work site captured by the imaging device 50.
- the real image RG is a two-dimensional image.
- the remote control system 100 includes a remote control device 70 provided at a remote control facility existing outside the hydraulic shovel 1, and a display system 200 for displaying an image of the object WAa and the transport vehicle WAb present at the work site at the remote control facility. Equipped with
- the display system 200 includes a display device 80 existing outside the hydraulic shovel 1 and a control device 90 existing outside the hydraulic shovel 1 and capable of communicating with the hydraulic shovel 1.
- Each of the remote control device 70, the display device 80, and the control device 90 is provided at a remote control facility.
- Each of the remote control device 70, the display device 80, and the control device 90 is provided separately from the hydraulic shovel 1 and the transporter vehicle WAb.
- the communication system 300 may perform wired communication or wireless communication.
- the communication system 300 includes at least one of the Internet, a Local Area Network (LAN), a mobile telephone communication network, and a satellite communication network.
- the hydraulic shovel 1 is remotely controlled by the remote controller 70.
- the remote control device 70 includes a working lever 71 for remotely operating the swing structure 3 and the working machine 4 of the hydraulic shovel 1 and a traveling lever 72 for remotely operating the traveling body 2.
- the operator operates the remote control device 70 at the remote control facility.
- An operation signal generated by operating the remote control device 70 is transmitted to the control device 60 via the communication system 300.
- the control device 60 outputs a control signal for controlling the swing body 3, the work implement 4, and the traveling body 2 based on the operation signal. Thereby, the hydraulic shovel 1 is remotely operated.
- the work lever 71 includes a left work lever 71 ⁇ / b> L and a right work lever 71 ⁇ / b> R operated to operate the swing body 3 and the work machine 4.
- the traveling lever 72 includes a left traveling lever 72L and a right traveling lever 72R which are operated to operate the traveling body 2.
- the transport vehicle WAb operates at the work site.
- the transport vehicle WAb is a dump truck, and a loading platform 401, a vehicle body 402 supporting the loading platform 401, a traveling device 403 supporting the vehicle body 402, a position detection device 430 (second position detection device), and a posture detection device 440 (Second posture detection device) and a control device 460.
- the traveling device 403 includes a wheel supporting a tire and a steering device.
- the driver gets on the transport vehicle WAb.
- the transport vehicle WAb is operated by the driver.
- the hydraulic shovel 1 excavates the construction target with the working machine 4 and loads the load generated by the excavation into the transport vehicle WAb.
- the transport vehicle WAb transports the load loaded by the hydraulic shovel 1.
- the transport vehicle WAb may be, for example, an unmanned transport vehicle (unmanned dump truck) that travels based on a command signal transmitted from the management system.
- unmanned transport vehicle unmanned dump truck
- a vehicle body coordinate system (Xn-Yn-Zn coordinate system) is defined in the vehicle body 402.
- the vehicle body coordinate system is a three-dimensional coordinate system that indicates the relative position with respect to the origin PN defined on the vehicle body 402.
- the vehicle body coordinate system of the vehicle body 402 is defined by the Xn axis of the fourth reference plane with respect to the origin PN, the Yn axis orthogonal to the Xn axis in the fourth reference plane, and the Zn axis orthogonal to the fourth reference plane. Ru.
- the direction parallel to the Xn axis is taken as the Xn axis direction
- the direction parallel to the Yn axis is taken as the Yn axis direction
- the direction parallel to the Zn axis is taken as the Zn axis direction.
- the Xn axis direction is the longitudinal direction of the vehicle body 402
- the Yn axis direction is the lateral direction of the vehicle body 402
- the Zn axis direction is the vertical direction of the vehicle body 402.
- the position detection device 430 is mounted on the transport vehicle WAb.
- the position detection device 430 detects the position Pb of the transport vehicle WAb in the global coordinate system (Xg-Yg-Zg coordinate system).
- the position detection device 430 is provided on the vehicle body 402.
- the position detection device 430 detects the position Pb of the vehicle body 402 using the GNSS. Detection data of the position detection device 430 is output to the control device 460.
- Posture detection device 440 is mounted on transport vehicle WAb.
- the attitude detection device 440 detects the attitude of the transport vehicle WAb.
- the attitude of the transport vehicle WAb includes the attitude of the vehicle body 402.
- the posture of the vehicle body 402 includes the inclination angle of the vehicle body 402.
- Detection data of the posture detection device 440 is output to the control device 460.
- the posture detection device 440 includes an inertial measurement unit (IMU) provided in the vehicle body 402.
- IMU inertial measurement unit
- FIG. 2 is a side view schematically showing the hydraulic shovel 1 according to the present embodiment.
- FIG. 3 is a rear view schematically showing the hydraulic shovel 1 according to the present embodiment.
- FIG. 4 is a plan view schematically showing the hydraulic shovel 1 according to the present embodiment.
- the hydraulic shovel 1 includes a distance detection device 20, a position detection device 30, a posture detection device 40 including a swing body posture sensor 41 and a work machine posture sensor 42; An imaging device 50 and a control device 60 are included.
- the revolving unit 3 is provided with a plurality of GPS antennas 31.
- the GPS antenna 31 receives a radio wave from a GPS satellite and outputs a signal generated based on the received radio wave to the position detection device 30.
- the position detection device 30 detects the position Pr at which the GPS antenna 31 defined in the global coordinate system is installed, based on the signal from the GPS antenna 31.
- the position detection device 30 detects the position Pa of the revolving unit 3 in the global coordinate system based on the position Pr at which the GPS antenna 31 is installed.
- the position detection device 30 detects each of a position Pra where one GPS antenna 31 is installed and a position Prb where the other GPS antenna 31 is installed.
- the position detection device 30 performs arithmetic processing based on at least one of the position Pra and the position Prb to detect the position Pa of the revolving unit 3 in the global coordinate system.
- the position Pa of the swing body 3 is a position Pra.
- Position Pa of revolving unit 3 may be position Prb or may be a position between position Pra and position Prb.
- the rotating body posture sensor 41 includes an inertial measurement unit (IMU).
- the rotating body posture sensor 41 detects the inclination angle of the rotating body 3 with respect to the Xg-Yg plane (horizontal plane) defined by the global coordinate system.
- the tilt angle of the swing body 3 is centered at the roll angle ⁇ 1 indicating the tilt angle of the swing body 3 in the left-right direction with respect to the horizontal plane, the pitch angle ⁇ 2 indicating the tilt angle of the swing body 3 in the front-back direction with respect to the horizontal plane, and the Zm axis And Yaw angle ⁇ 3 indicating the inclination angle of the swing body 3 in the rotational direction.
- the roll angle ⁇ 1, the pitch angle ⁇ 2, and the yaw angle ⁇ 3 are calculated by time-integrating the angular velocity detected by the inertial measurement device.
- the yaw angle ⁇ 3 changes.
- the swing body 3 turns while the traveling body 2 is stopped, the yaw angle ⁇ 3 corresponds to the swing angle of the swing body 3.
- the posture of the revolving unit 3 includes at least one of a roll angle ⁇ 1, a pitch angle ⁇ 2, and a yaw angle ⁇ 3.
- the yaw angle ⁇ 3 may be detected by the position detection device 30.
- the position detection device 30 can detect the orientation of the swing body 3 with respect to the reference orientation in the global coordinate system based on the position Pra of one GPS antenna 31 and the position Prb of the other GPS antenna 31.
- the position detection device 30 performs arithmetic processing based on the position Pra and the position Prb to detect the orientation of the swing body 3 with respect to the reference orientation.
- the angle between the reference orientation and the orientation of the swing body 3 corresponds to the yaw angle ⁇ 3.
- the position detection device 30 can calculate a straight line connecting the position Pra and the position Prb, and detect the yaw angle ⁇ 3 based on the angle formed by the calculated straight line and the reference azimuth.
- Work implement attitude sensor 42 includes a stroke sensor.
- the work implement attitude sensor 42 detects an inclination angle of the work implement 4 defined by the vehicle body coordinate system.
- the stroke sensor includes a boom stroke sensor 42A that detects the cylinder stroke of the boom cylinder 10, an arm stroke sensor 42B that detects the cylinder stroke of the arm cylinder 11, and a bucket stroke sensor 42C that detects the cylinder stroke of the bucket cylinder 12.
- Control device 60 calculates inclination angle ⁇ of boom 6 with respect to the Zm axis of the vehicle body coordinate system based on the detection data of boom stroke sensor 42A.
- the control device 60 calculates the inclination angle ⁇ of the arm 7 with respect to the boom 6 based on the detection data of the arm stroke sensor 42B.
- the control device 60 calculates the inclination angle ⁇ of the bucket 8 with respect to the arm 7 based on the detection data of the bucket stroke sensor 42C.
- the posture of the work implement 4 includes at least one of the inclination angle ⁇ , the inclination angle ⁇ , and the inclination angle ⁇ .
- FIG. 5 is a side view schematically showing the transport vehicle WAb according to the present embodiment.
- FIG. 6 is a rear view schematically showing the transport vehicle WAb according to the present embodiment.
- FIG. 7 is a plan view schematically showing the transport vehicle WAb according to the present embodiment.
- the transport vehicle WAb has a position detection device 430, a posture detection device 440, and a control device 460.
- a plurality of GPS antennas 431 are provided on the vehicle body 402.
- the position detection device 430 detects the position where the GPS antenna 431 defined in the global coordinate system is installed, based on the signal from the GPS antenna 431.
- the position detection device 430 detects the position Pb of the vehicle body 402 in the global coordinate system based on the position where the GPS antenna 431 is installed.
- Posture detection device 440 includes an inertial measurement device (IMU).
- the posture detection device 440 detects the inclination angle of the vehicle body 402 with respect to the Xg-Yg plane (horizontal plane) defined by the global coordinate system.
- the tilt angle of the vehicle body 402 is a roll angle ⁇ 4 indicating the tilt angle of the vehicle body 402 in the left-right direction with respect to the horizontal plane, a pitch angle ⁇ 5 indicating the tilt angle of the vehicle body 402 in the front-rear direction with respect to the horizontal plane, and a rotational direction about the Zn axis And a yaw angle ⁇ 6 indicating the inclination angle of the vehicle body 402.
- the roll angle ⁇ 4, the pitch angle ⁇ 5, and the yaw angle ⁇ 6 are calculated by temporally integrating the angular velocity detected by the inertial measurement device.
- the yaw angle ⁇ 6 may be detected by the position detection device 430. When the vehicle body 402 turns, the yaw angle ⁇ 6 changes.
- the posture of the vehicle body 402 includes at least one of a roll angle ⁇ 4, a pitch angle ⁇ 5, and a yaw angle ⁇ 6.
- the yaw angle ⁇ 6 changes.
- FIG. 8 is a view schematically showing the distance detection device 20 and the imaging device 50 according to the present embodiment.
- a global coordinate system Xg-Yg-Zg coordinate system
- a vehicle body coordinate system Xm-Ym-Zm coordinate system
- a distance detection device coordinate system (Xd-Yd-Zd coordinate system) and an imaging device coordinate system (Xc-Yc-Zc coordinate system) are defined.
- the distance detection device coordinate system is a three-dimensional coordinate system that indicates the relative position to the origin PD defined in the distance detection device 20.
- the distance detection device coordinate system is defined by the Xd axis of the second reference plane with respect to the origin PD, the Yd axis orthogonal to the Xd axis in the second reference plane, and the Zd axis orthogonal to the second reference plane .
- the origin PD of the distance detection device coordinate system is defined at the center of the detection surface 20P of the distance detection element 20RC.
- the imaging device coordinate system is a three-dimensional coordinate system that indicates the relative position with respect to the origin PC defined in the imaging device 50.
- the imaging device coordinate system is defined by the Xc axis of the third reference plane with respect to the origin PC, the Yc axis orthogonal to the Xc axis in the third reference plane, and the Zc axis orthogonal to the third reference plane.
- the origin PC of the imaging device coordinate system is defined at the center of the imaging surface 19P of the image sensor 19RC.
- the Xc axis passes through the optical center of the imaging device 50 and is orthogonal to the imaging surface 19P.
- the relative position between the position Pa of the swing body 3 in the global coordinate system detected by the position detection device 30 and the origin PL in the vehicle body coordinate system of the swing body 3 is derived from, for example, design data or specification data of the hydraulic shovel 1 Known data.
- the relative positions of the origin PL in the vehicle body coordinate system of the swing structure 3 and the origin PD in the distance detection device coordinate system and the origin PC in the imaging device coordinate system are, for example, design data of the hydraulic shovel 1 and design data of the distance detection device 20 Alternatively, it is known data derived from specification data, design data or specification data of the imaging device 50, or the like. Therefore, the position Pa of the swing body 3 in the global coordinate system, the position of the swing body 3 in the vehicle body coordinate system, the position in the distance detection device coordinate system, and the position in the imaging device coordinate system can be mutually converted.
- the relative position between the position Pb of the vehicle body 402 in the global coordinate system detected by the position detection device 430 and the origin PN in the vehicle body coordinate system of the vehicle body 402 is derived from, for example, design data or specification data of the transport vehicle WAb. Known data.
- the relative position between the swing body 3 and the vehicle body 402 in the global coordinate system is defined by the position Pa of the swing body 3 detected by the position detection device 30 and the position Pb of the vehicle body 402 detected by the position detection device 430. Ru. Therefore, the position of the swing body 3 in the global coordinate system, the position Pb of the vehicle body 402 in the global coordinate system, the position of the swing body 3 in the vehicle coordinate system, and the position Pb in the vehicle coordinate system of the vehicle body 402 are mutually converted. It is possible. Further, the position of the vehicle body 402 in the global coordinate system, the position of the vehicle body 402 in the vehicle body coordinate system, the position in the distance detection device coordinate system, and the position in the imaging device coordinate system can be mutually converted.
- FIG. 9 is a view schematically showing an example of the distance detection device 20 and the imaging device 50 according to the present embodiment.
- FIG. 9 shows an example in which the object WAa detected by the distance detection device 20 and imaged by the imaging device 50 is the ground surface of the construction object.
- the distance detection device 20 has a detection range MA.
- the distance detection device 20 detects the distance Ld to the object WAa present in the detection range MA.
- the distance Ld is the distance between the origin PD and the surface of the object WAa in the distance detection device coordinate system.
- the distance detection device 20 detects the distance Ld from the surface of the object WAa, and detects the position Pd of the surface of the object WAa in the distance detection device coordinate system.
- the position Pd defines the orientation and angle with respect to the origin PD.
- the distance detection device 20 can detect three-dimensional coordinates (Xd, Yd, Zd) of the position Pd in the distance detection device coordinate system.
- the distance detection device 20 detects the distance Ld from each of the plurality of portions on the surface of the object WAa, and detects the position Pd of each of the plurality of portions, whereby the three-dimensional shape of the object WAa in the distance detection coordinate system Can be detected.
- the three-dimensional shape of the object WAa is point cloud data of a plurality of positions Pd at which three-dimensional coordinates (Xd, Yd, Zd) are detected.
- the distance detection device 20 detects the three-dimensional shape of the object WAa in the forward direction of the rotating body 3.
- the distance detection device 20 can detect the distance Ld from each of the plurality of parts on the ground surface, and detect the topography of the hydraulic shovel 1 in the forward direction.
- the position in the global coordinate system, the position in the vehicle body coordinate system of the revolving structure 3, the position in the vehicle body coordinate system of the vehicle body 402, the position in the distance detection device coordinate system, and the position in the imaging device coordinate system mutually convertible. Therefore, the point group data of the plurality of positions Pd (Xd, Yd, Zd) defined in the distance detection device coordinate system are the plurality of positions Pm (Xm, Ym, Zm) defined in the vehicle body coordinate system of the swing body 3 Of the plurality of positions Pn (Xn, Yn, Zn) defined in the vehicle body coordinate system of the vehicle body 402, and is defined in the global coordinate system.
- the three-dimensional shape of the object WAa defined in the distance detection device coordinate system is coordinate-converted to the vehicle body coordinate system of the swing body 3, the vehicle body coordinate system of the vehicle body 402, the global coordinate system, and the imaging device coordinates. It is convertible to the three-dimensional shape of each object WAa of the system.
- the imaging device 50 has an imaging range TA.
- the imaging device 50 can acquire an image of the target WAa present in the imaging range TA.
- the detection range MA and at least a part of the imaging range TA overlap on the surface of the object WAa.
- the installation state of the distance detection device 20 with respect to the revolving unit 3 and the installation state of the imaging device 50 are adjusted such that the detection range MA and at least a part of the imaging range TA overlap.
- the overlapping range of the detection range MA and the imaging range TA is preferably as large as possible.
- the imaging device 50 acquires an image corresponding to each of the plurality of positions Pd.
- the position Pc in the image corresponding to the position Pd is defined in the imaging device coordinate system.
- the three-dimensional shape of the object WAa and the image of the object WAa in the imaging device coordinate system by converting the position Pd in the three-dimensional shape of the object WAa detected by the distance detection device 20 into the position Pc in the image of the imaging device coordinate system And can be superimposed. Further, the position Pd in the three-dimensional shape of the object WAa and the position Pc in the image of the object WAa are the position Pm of the vehicle body coordinate system of the revolving unit 3 or the position Pn of the vehicle body coordinate system of the vehicle body 402 or the position Pg of the global coordinate system. It is convertible.
- FIG. 10 is a functional block diagram showing an example of the control device 60, the control device 460, and the control device 90 according to the present embodiment.
- Control device 60 includes a computer system.
- the controller 60 includes an arithmetic processing unit 60A including a processor such as a CPU (Central Processing Unit), a volatile memory such as a RAM (Random Access Memory), and a non-volatile memory such as a ROM (Read Only Memory).
- Controller 460 includes a computer system.
- Control device 460 includes an arithmetic processing unit 460A including a processor such as a CPU (Central Processing Unit), a volatile memory such as a RAM (Random Access Memory), and a non-volatile memory such as a ROM (Read Only Memory).
- Controller 90 includes a computer system.
- the controller 90 includes an arithmetic processing unit 90A including a processor such as a CPU (Central Processing Unit), a volatile memory such as a RAM (Random Access Memory), and a non-volatile memory such as a ROM (Read Only Memory). It has a storage device 90B and an input / output interface 90C.
- a processor such as a CPU (Central Processing Unit)
- a volatile memory such as a RAM (Random Access Memory)
- ROM Read Only Memory
- the display device 80 includes a flat panel display such as a liquid crystal display (LCD) or an organic electroluminescence display (OELD).
- a flat panel display such as a liquid crystal display (LCD) or an organic electroluminescence display (OELD).
- LCD liquid crystal display
- OELD organic electroluminescence display
- Detection data indicating the distance Ld to the object WAa detected by the distance detection device 20 is output to the control device 60 at a predetermined cycle.
- the input / output interface 60C of the control device 60 receives detection data of the distance detection device 20.
- the input / output interface 60C of the control device 60 transmits detection data of the distance detection device 20 to the control device 90 at a predetermined cycle via the communication system 300.
- Detection data indicating the position Pa of the swing body 3 detected by the position detection device 30 is output to the control device 60 at a predetermined cycle.
- the input / output interface 60C of the control device 60 receives detection data of the position detection device 30.
- the input / output interface 60C of the control device 60 transmits detection data of the position detection device 30 to the control device 90 at a predetermined cycle via the communication system 300.
- Detection data indicating the posture of the hydraulic shovel 1 detected by the posture detection device 40 is output to the control device 60 at a predetermined cycle.
- the input / output interface 60C of the control device 60 receives detection data of the posture detection device 40.
- the input / output interface 60C of the control device 60 transmits detection data of the posture detection device 40 to the control device 90 at a predetermined cycle via the communication system 300.
- Image data of the target WAa captured by the imaging device 50 is output to the control device 60 at a predetermined cycle.
- the input / output interface 60C of the control device 60 receives the image data of the imaging device 50.
- the input / output interface 60C of the control device 60 transmits the image data of the object WAa to the control device 90 at a predetermined cycle via the communication system 300.
- Detection data indicating the position Pb of the vehicle body 402 detected by the position detection device 430 is output to the control device 460 at a predetermined cycle.
- the input / output interface 460C of the controller 460 receives detection data of the position detector 430.
- the input / output interface 460C of the control device 460 transmits detection data of the position detection device 430 to the control device 90 at a predetermined cycle via the communication system 300.
- Detection data indicating the attitude of the transport vehicle WAb detected by the attitude detection device 440 is output to the control device 460 at a predetermined cycle.
- An input / output interface 460C of the control device 460 receives detection data of the posture detection device 440.
- the input / output interface 460C of the control device 460 transmits detection data of the posture detection device 440 to the control device 90 at a predetermined cycle via the communication system 300.
- the control device 90 detects detection data of the distance detection device 20, detection data of the position detection device 30, detection data of the posture detection device 40, image data of the object WAa captured by the imaging device 50, position detection via the communication system 300.
- the detection data of the device 430 and the detection data of the posture detection device 440 are received at a predetermined cycle.
- the control device 90 generates a target image AGa showing a virtual viewpoint image of the target WAa viewed from a virtual viewpoint outside the hydraulic shovel 1 at a predetermined cycle, based on detection data of the distance detection device 20.
- control device 90 is a working machine image showing a virtual viewpoint image of the hydraulic shovel 1 viewed from an external virtual viewpoint of the hydraulic shovel 1 based on the detection data of the position detection device 30 and the detection data of the posture detection device 40 BG is generated at a predetermined cycle. Further, based on the detection data of the position detection device 430 and the detection data of the posture detection device 440, the control device 90 is a virtual viewpoint image of the transport vehicle WAb viewed from a virtual viewpoint outside the hydraulic shovel 1 and the transport vehicle WAb. A transport vehicle image AGb shown is generated at a predetermined cycle. Further, the control device 90 generates a composite image IG in which the target image AGa, the work machine image BG, and the transport vehicle image AGb are superimposed at a predetermined cycle.
- the display device 80 displays the composite image IG generated by the control device 90.
- the composite image IG including the target image AGa, the work machine image BG, and the transport vehicle image AGb is computer graphics (computer graphics) generated by the controller 90.
- Each of the target image AGa, the work machine image BG, and the transport vehicle image AGb displayed on the display device 80 is a three-dimensional image viewed from an external virtual viewpoint of the hydraulic shovel 1 and the transport vehicle WAb.
- the display device 80 also displays an actual image RG indicating an actual image of the target WAa at the work site captured by the imaging device 50.
- the real image RG is a two-dimensional image.
- the virtual viewpoint means a viewpoint virtually set in a virtual space simulating a real space (in the present embodiment, a work site).
- the position and orientation of the virtual viewpoint can be set arbitrarily.
- the virtual viewpoint image refers to a virtual image viewed from a virtual viewpoint whose position and orientation are set in a virtual space.
- a virtual viewpoint image refers to a virtual image that is captured when capturing an image of a virtual space with an imaging device that is installed at a virtual viewpoint and has its position and orientation set in the virtual space.
- the virtual viewpoint image is computer graphics generated by the controller 90.
- the position of the virtual viewpoint is set outside the hydraulic shovel 1 and the transport vehicle WAb and obliquely upward of the hydraulic shovel 1 and the transport vehicle WAb.
- the direction of the virtual viewpoint is set obliquely downward so as to look down on the hydraulic shovel 1, the target WAa around the hydraulic shovel 1, and the transport vehicle WAb.
- the target image AGa is a virtual three-dimensional image of the target WAa viewed from the virtual viewpoint in the virtual space of the work site.
- the work machine image BG is a virtual three-dimensional image of the hydraulic shovel 1 viewed from a virtual viewpoint in the virtual space of the work site.
- the transport vehicle image AGb is a virtual three-dimensional image of the transport vehicle WAb viewed from the virtual viewpoint in the virtual space of the work site.
- the arithmetic processing unit 90A includes a data receiving unit 97, a target image generating unit 91, a work machine image generating unit 92, a transport vehicle image generating unit 95, a combining unit 96, and a display control unit 93.
- the storage device 90B includes a coordinate conversion data storage unit 94.
- the data receiving unit 97 receives the detection data of the distance detection device 20, the detection data of the position detection device 30, the detection data of the posture detection device 40, and the image data of the imaging device 50 via the communication system 300. Also, the data receiving unit 97 receives detection data of the position detection device 430 and detection data of the attitude detection device 440 via the communication system 300.
- the target image generation unit 91 generates a target image AGa indicating a virtual viewpoint image of the target WAa viewed from a virtual viewpoint outside the hydraulic shovel 1 at a predetermined cycle, based on detection data of the distance detection device 20.
- the detection data of the distance detection device 20 includes the three-dimensional shape of the surface of the object WAa.
- the target image AGa is a three-dimensional image of the target WAa viewed from a virtual viewpoint.
- the target image generation unit 91 generates a target image AGa that is a three-dimensional image of the target WAa viewed from the virtual viewpoint based on the three-dimensional shape of the target WAa.
- the target image generation unit 91 generates the target image AGa defined by the distance detection device coordinate system at a predetermined cycle.
- the target image AGa is a construction target (terrain)
- the three-dimensional shape of the target WAa changes at the work site due to excavating work, ground leveling work, and the like.
- the target image generation unit 91 generates a target image AGa at a predetermined cycle.
- the object image generation unit 91 When the three-dimensional shape of the object WAa is changed due to an excavation operation, a ground adjustment operation, or the like, the object image generation unit 91 generates an object image AGa linked to the change of the three-dimensional shape of the object WAa.
- the working machine image generation unit 92 is a working machine image showing a virtual viewpoint image of the hydraulic shovel 1 viewed from an external virtual viewpoint of the hydraulic shovel 1 based on the detection data of the position detection device 30 and the detection data of the posture detection device 40 BG is generated at a predetermined cycle.
- Detection data of the posture detection device 40 includes the posture of the swing body 3 of the hydraulic shovel 1 and the posture of the work implement 4.
- the work machine image BG is a three-dimensional image of the hydraulic shovel 1 viewed from a virtual viewpoint.
- the work machine image generation unit 92 generates a work machine image BG, which is a three-dimensional image of the hydraulic shovel 1 viewed from a virtual viewpoint, at a predetermined cycle, based on the posture of the swing body 3 and the posture of the work machine 4.
- the work machine image generation unit 92 generates a work machine image BG defined by the vehicle body coordinate system of the revolving unit 3 at a predetermined cycle. Further, for example, when the swing body 3 turns and the yaw angle ⁇ 3 changes, the working machine image generation unit 92 generates the working machine image BG at a predetermined cycle so as to interlock with the turning of the swing body 3. Similarly, when at least one of the roll angle ⁇ 1 and the pitch angle ⁇ 2 changes, the work machine image generation unit 92 determines the work machine image BG to interlock with the change of at least one of the roll angle ⁇ 1 and the pitch angle ⁇ 2.
- the work machine image generation unit 92 interlocks with at least one change of the inclination angles ⁇ , ⁇ , and ⁇ .
- the work machine image BG is generated at a predetermined cycle.
- the conveyance vehicle image generation unit 95 Based on the detection data of the position detection device 430 and the detection data of the posture detection device 440, the conveyance vehicle image generation unit 95 generates a virtual viewpoint image of the conveyance vehicle WAb viewed from a virtual viewpoint outside the hydraulic shovel 1 and the conveyance vehicle WAb.
- a transport vehicle image AGb shown is generated at a predetermined cycle.
- the transport vehicle image AGb is a three-dimensional image of the transport vehicle WAb viewed from a virtual viewpoint.
- the transportation vehicle image generation unit 95 generates a transportation vehicle image AGb, which is a three-dimensional image of the transportation vehicle WAb viewed from the virtual viewpoint, at a predetermined cycle, based on the posture of the transportation vehicle WAb.
- the transport vehicle image generation unit 95 generates a transport vehicle image AGb defined by the vehicle body coordinate system of the vehicle body 402 at a predetermined cycle. Further, for example, when the transport vehicle WAb turns and the yaw angle ⁇ 6 changes, the transport vehicle image generation unit 95 generates the transport vehicle image AGb at a predetermined cycle so as to interlock with the turning of the vehicle body 402. Similarly, when at least one of the roll angle ⁇ 4 and the pitch angle ⁇ 5 changes, the transport vehicle image generation unit 95 determines the transport vehicle image AGb so as to interlock with the change of at least one of the roll angle ⁇ 4 and the pitch angle ⁇ 5.
- the transport vehicle image generation unit 95 performs transportation so as to interlock with the change of the three-dimensional shape or position of the transport vehicle WAb.
- a vehicle image AGb is generated.
- the combining unit 96 combines the work machine image BG generated by the work machine image generation unit 92 and the transport vehicle image AGb generated by the transport vehicle image generation unit 95 to generate a composite image IG.
- the combining unit 96 generates the combined image IG at a predetermined cycle.
- the composite image IG is an image in which the work machine image BG and the transport vehicle image AGb are superimposed.
- the combining unit 96 generates the target image AGa generated by the target image generation unit 91, the work machine image BG generated by the work machine image generation unit 92, and the transport vehicle image generation unit 95.
- the transport vehicle image AGb is synthesized.
- the composite image IG is an image in which the target image AGa, the work machine image BG, and the transport vehicle image AGb are superimposed.
- the target image AGa may not be included in the composite image IG.
- the display control unit 93 causes the display device 80 to display the composite image IG.
- the display control unit 93 acquires from the combining unit 96 the combined image IG generated by the combining unit 96.
- the display control unit 93 converts the composite image IG into display data that can be displayed by the display device 80, and outputs the display data to the display device 80. As a result, the composite image IG is displayed on the display device 80.
- the coordinate conversion data storage unit 94 includes the position in the global coordinate system, the position in the vehicle body coordinate system of the revolving unit 3, the position in the vehicle body coordinate system of the vehicle body 402, the position in the distance detection device coordinate system, and the image pickup device coordinate system. It stores coordinate conversion data for mutually converting position and position. Coordinate conversion data includes coordinate conversion parameters or coordinate conversion determinants.
- the position Pa of the rotating body 3 in the global coordinate system, the position Pb of the vehicle body 402 in the global coordinate system, the origin PL in the vehicle body coordinate system of the rotating body 3, the origin PN in the vehicle body coordinate system of the vehicle body 402, and the distance detection device coordinates The relative position between the origin PD in the system and the origin PC in the imaging device coordinate system is known data derived from design data or specification data. Therefore, coordinate conversion data can be derived in advance based on known data.
- the coordinate conversion data storage unit 94 stores coordinate conversion data derived in advance.
- the combining unit 96 generates the target image AGa, and the work such that the combined image IG is displayed on the display device 80 in a predetermined single coordinate system based on the coordinate conversion data stored in the coordinate conversion data storage unit 94. Coordinate transformation is performed on at least one of the machine image BG and the transport vehicle image AGb.
- the display control unit 93 causes the display device 80 to display the composite image IG in a single coordinate system.
- the target image generation unit 91 can generate the target image AGa in the distance detection device coordinate system based on the detection data of the distance detection device 20.
- the work machine image generation unit 92 can generate the work machine image BG in the vehicle body coordinate system of the swingable body 3 based on the detection data of the posture detection device 40. Further, the work machine image generation unit 92 can generate the work machine image BG in the global coordinate system based on the detection data of the position detection device 30 and the detection data of the posture detection device 40.
- the transport vehicle image generation unit 95 can generate the transport vehicle image AGb in the vehicle body coordinate system of the vehicle body 402 based on the detection data of the posture detection device 440.
- Transport vehicle image generation unit 95 can generate transport vehicle image AGb in the global coordinate system based on detection data of position detection device 430 and detection data of posture detection device 440.
- the synthesis unit 96 is configured to display the coordinate system of the target image AGa, the coordinate system of the work machine image BG, and the transport vehicle such that the target image AGa, the work machine image BG, and the transport vehicle image AGb are displayed in a single coordinate system. Adjust at least one of the coordinate systems of the image AGb.
- the combining unit 96 is stored in the coordinate conversion data storage unit 94
- the target image AGa defined in the distance detection coordinate system is converted into the target image AGa defined in the vehicle body coordinate system of the rotating body 3 based on the coordinate conversion data.
- the combining unit 96 determines the transport vehicle image AGb defined in the vehicle body coordinate system of the vehicle body 402 or the global coordinate system as the vehicle body of the revolving vehicle 3. The image is converted into a transport vehicle image AGb defined by the coordinate system.
- the target image AGa, the work machine image BG, and the transport vehicle image AGb are defined in the vehicle body coordinate system of the revolving unit 3.
- the display control unit 93 causes the display device 80 to display the composite image IG including the target image AGa, the work machine image BG, and the transport vehicle image AGb defined in the vehicle body coordinate system of the revolving unit 3.
- the combining unit 96 stores the coordinates stored in the coordinate conversion data storage unit 94. Based on the conversion data, the target image AGa defined in the distance detection coordinate system is converted into the target image AGa defined in the global coordinate system, and the work machine image BG defined in the vehicle body coordinate system of the revolving unit 3 is obtained.
- the work vehicle image Bg defined in the global coordinate system is converted, and the transported vehicle image AGb defined in the vehicle body coordinate system of the vehicle body 402 is converted into a transported vehicle image AGb defined in the global coordinate system.
- the display control unit 93 causes the display device 80 to display a composite image IG including the target image AGa, the work machine image BG, and the transport vehicle image AGb defined in the global coordinate system.
- the display control unit 93 is a composite image IG including an actual image RG that is an actual image of the object WAa captured by the imaging device 50, a target image AGa, a work machine image BG, and a transport vehicle image AGb. And are simultaneously displayed on the display device 80.
- the display control unit 93 causes the composite image IG to be displayed on a part of the display screen on which the real image RG is displayed.
- FIG. 11 is a view showing a display example of the display device 80 according to the present embodiment.
- the imaging device 50 acquires an actual image RG which is an actual image of the target WAa in the forward direction of the revolving unit 3.
- the display control unit 93 displays the actual image RG of the work site acquired by the imaging device 50 on the display screen of the display device 80.
- the imaging device 50 acquires a real image RG of the target WAa in the forward direction of the revolving unit 3. In the imaging of the imaging device 50, at least a part of the work implement 4 is reflected.
- the actual image RG acquired by the imaging device 50 includes the actual image of the work implement 4.
- the display screen of the display device 80 that displays the actual image RG of the work site includes the actual image of the work implement 4.
- the actual image of the work implement 4 is displayed on the display screen of the display device 80 on which the actual image RG of the work site imaged by the imaging device 50 is displayed.
- the display control unit 93 combines the target image AGa, the work machine image BG, and the transport vehicle image AGb on a part of the display screen of the display device 80 that displays the actual image RG of the work site captured by the imaging device 50. Display the image IG simultaneously. That is, the display control unit 93 imposes the target image AGa, the work machine image BG, and the transport vehicle image AGb on a part of the display screen of the display device 80 that displays the actual image RG of the work site.
- the composite image IG including at least the work machine image BG and the transport vehicle image AGb, which is imposed on a part of the display screen of the display device 80 is appropriately referred to as an imposed image IG.
- the impose image IG includes not only the work machine image BG and the transport vehicle image AGb but also the target image AGa.
- the target image AGa, the work machine image BG, and the transport vehicle image AGb are displayed in a superimposed manner.
- the display control unit 93 causes the display screen to display the real image of the work machine 4 and the pose image IG including the work machine image BG and the transported vehicle image AGb at different positions.
- the actual image of the work implement 4 is displayed on the central portion and the right portion of the display screen of the display device 80.
- the display control unit 93 displays the pose image IG on the corner of the display screen of the display device 80 so that the real image of the work implement 4 and the pose image IG do not overlap.
- not only the actual image of the work implement 4 but also the actual image of the ground surface which is the construction target of the work implement 4 is displayed on the display screen of the display device 80.
- the display control unit 93 causes the pose image IG to be displayed in the upper left corner of the display screen of the display device 80 so that the pose image IG and the actual image of the work implement 4 and the ground surface do not overlap.
- the pose image IG may be displayed in the upper right corner of the display screen of the display device 80. Since the real image of the work machine 4 and the real image of the construction target are not hidden by the pose image IG, the operator smoothly carries out the work of constructing the construction target using the work machine 4 while looking at the real image. Can.
- the virtual viewpoint is set obliquely upward of the hydraulic shovel 1 and the target WAa around the hydraulic shovel 1.
- the ground surface to be constructed exists as the target WAa. That is, in the present embodiment, the target image AGa is a three-dimensional image showing a three-dimensional shape of the ground surface WAa which is a construction target viewed from a virtual viewpoint.
- the display control unit 93 displays the target image AGa interlocked with the change in the shape of the ground surface based on detection data of the distance detection device 20 Display on 80
- the work machine image BG is a skeleton image that simply simulates the hydraulic shovel 1 viewed from a virtual viewpoint.
- the working machine image BG moves in real time. That is, the work machine image BG moves in synchronization with the change in the position and attitude of the hydraulic shovel 1.
- the display control unit 93 detects that the hydraulic shovel 1 (traveling body 2) is moving based on the detection data of the position detection device 30, the display control unit 93 performs an operation to interlock with the change in the position of the hydraulic shovel 1.
- the machine image BG is moved.
- the working machine image BG is a traveling body image 2G which is a virtual viewpoint image simulating the traveling body 2, a swinging body image 3 G which is a virtual viewpoint image simulating the swinging body 3, and a virtual viewpoint image simulating the working machine 4.
- a work implement image 4G includes a boom image 6G that simulates the boom 6, an arm image 7G that simulates the arm 7, and a bucket image 8G that simulates the bucket 8.
- the traveling body image 2G includes a skeleton image that simply shows the outer shape of the traveling body 2.
- the revolving unit image 3G includes a skeleton image simply showing the outer shape of the revolving unit 3.
- Work implement image 4G includes a skeleton image simply showing the outer shape of work implement 4.
- the working machine image BG including the traveling body image 2G, the rotating body image 3G, and the working machine image 4G is three-dimensional computer graphics.
- the work machine image generation unit 92 acquires a skeleton image (three-dimensional CG model) stored in the storage device 90B from the storage device 90B, and based on the detection data of the position detection device 30 and the detection data of the posture detection device 40. , And generates a working machine image BG that moves in synchronization with the change in position and posture of the hydraulic shovel 1.
- the position detection device 30 transmits detection data indicating that the hydraulic shovel 1 has moved to the control device 90 at a predetermined cycle.
- the work machine image generation unit 92 moves the work machine image BG so as to be interlocked with the movement of the hydraulic shovel 1.
- the display control unit 93 causes the display device 80 to display the work machine image BG so that the work machine image BG (traveling body image 2G) moves.
- the display control unit 93 moves the working machine image BG to be interlocked with the change in the attitude of the hydraulic shovel 1 based on the detection data of the attitude detection device 40. .
- the swing body posture sensor 41 transmits detection data indicating that the swing body 3 has turned to the control device 90 at a predetermined cycle.
- the working machine image generation unit 92 turns the turning body image 3G of the working machine image BG so as to interlock with the turning of the turning body 3 of the hydraulic shovel 1.
- the display control unit 93 causes the display device 80 to display the work machine image BG so that the turning body image 3G is turned.
- the work machine posture sensor 42 transmits detection data indicating that the work machine 4 has operated to the control device 90.
- the work machine image generation unit 92 operates the work machine image 4G of the work machine image BG so as to be interlocked with the operation of the work machine 4 of the hydraulic shovel 1.
- the display control unit 93 causes the display device 80 to display the work machine image BG so that the work machine image 4G operates. For example, the display control unit 93 moves the boom image 6G to interlock with the change of the inclination angle ⁇ , moves the arm image 7G to interlock with the change of the inclination angle ⁇ , and interlocks with the change of the inclination angle Move the bucket image 8G.
- the transport vehicle image AGb is a skeleton image that simulates the transport vehicle WAb simply viewed from the virtual viewpoint. According to the movement of the transport vehicle WAb and the change of the attitude of the transport vehicle WAb, the transport vehicle image AGb moves in real time. That is, the transport vehicle image AGb moves in synchronization with changes in the position and attitude of the transport vehicle WAb.
- the display control unit 93 detects that the transport vehicle WAb is moving based on the detection data of the position detection device 430
- the display control unit 93 moves the transport vehicle image AGb to interlock with the change in the position of the transport vehicle WAb.
- the position detection device 430 transmits detection data indicating that the transport vehicle WAb has moved to the control device 90.
- the transport vehicle image generation unit 95 moves the transport vehicle image AGb so as to be interlocked with the movement of the transport vehicle WAb.
- the display control unit 93 causes the display device 80 to display the transport vehicle image AGb so that the transport vehicle image AGb moves.
- Transport vehicle image AGb includes a skeleton image that simply shows the outer shape of transport vehicle WAb.
- the transport vehicle image AGb is three-dimensional computer graphics.
- Transport vehicle image generation unit 95 acquires a skeleton image (three-dimensional CG model) stored in storage device 90 B from storage device 90 B, and based on the detection data of position detection device 430 and the detection data of attitude detection device 440 , And generates a transport vehicle image AGb that moves in synchronization with the change in position and posture of the hydraulic shovel 1.
- the display control unit 93 moves the transport vehicle image AGb based on the detection data of the attitude detection device 440 to interlock with the change in the attitude of the transport vehicle WAb.
- the posture detection device 440 transmits detection data indicating that the transport vehicle WAb has inclined or turned to the control device 90.
- the transport vehicle image generation unit 95 tilts or turns the transport vehicle image AGb so as to be interlocked with the tilt or turn of the transport vehicle WAb.
- the display control unit 93 causes the display device 80 to display the transport vehicle image AGb such that the transport vehicle image AGb tilts or turns.
- the shape of the transport vehicle WAb changes.
- the distance detection device 20 can detect the shape of the transport vehicle WAb.
- the display control unit 93 is a transport vehicle image AGb that is a character image so as to be interlocked with the change of the shape of the transport vehicle WAb based on detection data of the distance detection device 20.
- the shape or display form of may be changed.
- FIGS. 12, 13, and 14 are flowcharts showing an example of the display method according to the present embodiment.
- FIG. 12 is a flowchart showing the processing of the control device 60 of the hydraulic shovel 1.
- FIG. 13 is a flowchart showing processing of the control device 460 of the transport vehicle WAb.
- FIG. 14 is a flowchart showing processing of the control device 90 of the remote control system 100. The processing shown in FIGS. 12, 13 and 14 is performed at a predetermined cycle.
- the distance detection device 20 detects the distance Ld to the object WAa, and detects the three-dimensional shape of the object WAa.
- Position detection device 30 detects position Pa of revolving unit 3.
- the posture detection device 40 detects the posture of the hydraulic shovel 1 including the posture of the swing body 3 and the posture of the work implement 4.
- the posture of the revolving unit 3 includes a roll angle ⁇ 1, a pitch angle ⁇ 2, and a yaw angle ⁇ 3.
- the yaw angle ⁇ 3 changes.
- the yaw angle ⁇ 3 corresponds to the swing angle of the swing body 3.
- the posture of the work implement 4 includes the inclination angle ⁇ , the inclination angle ⁇ , and the inclination angle ⁇ .
- the imaging device 50 acquires image data indicating the actual image RG of the work site.
- Detection data of the distance detection device 20 detection data of the position detection device 30, detection data of the posture detection device 40, and image data of the imaging device 50 are output to the control device 60.
- the input / output interface 60C of the control device 60 acquires detection data of the distance detection device 20 from the distance detection device 20 (step S10).
- the input / output interface 60C of the control device 60 acquires detection data of the position detection device 30 from the position detection device 30 (step S20).
- the input / output interface 60C of the control device 60 acquires detection data of the posture detection device 40 from the posture detection device 40 (step S30).
- the input / output interface 60C of the control device 60 acquires the image data of the imaging device 50 from the imaging device 50 (step S40).
- step S10, step S20, step S30, and step S40 are performed is arbitrary, and may be performed simultaneously.
- the input / output interface 60C of the control device 60 transmits detection data of the distance detection device 20 to the control device 90 via the communication system 300 (step S50).
- the input / output interface 60C of the control device 60 transmits detection data of the position detection device 30 to the control device 90 via the communication system 300 (step S60).
- the input / output interface 60C of the control device 60 transmits the detection data of the posture detection device 40 to the control device 90 via the communication system 300 (step S70).
- the input / output interface 60C of the control device 60 transmits the image data of the imaging device 50 to the control device 90 via the communication system 300 (step S80).
- step S50, step S60, step S70, and step S80 are performed is arbitrary, and may be performed simultaneously.
- the position detection device 430 detects the position Pb of the vehicle body 402. Detection data of the position detection device 430 is output to the control device 460.
- the posture detection device 440 detects the posture of the vehicle body 402.
- the posture of the vehicle body 402 includes a roll angle ⁇ 4, a pitch angle ⁇ 5, and a yaw angle ⁇ 6. Detection data of the posture detection device 440 is output to the control device 460.
- the input / output interface 460C of the control device 460 acquires detection data of the position detection device 430 from the position detection device 430 (step S110).
- the input / output interface 460C of the control device 460 acquires detection data of the posture detection device 440 from the posture detection device 440 (step S120).
- step S110 and step S120 are performed are arbitrary, and may be performed simultaneously.
- the input / output interface 460C of the control device 460 transmits the detection data of the position detection device 430 to the control device 90 via the communication system 300 (step S130).
- the input / output interface 460C of the control device 460 transmits detection data of the posture detection device 440 to the control device 90 via the communication system 300 (step S140).
- step S130 and step S140 are performed are arbitrary, and may be performed simultaneously.
- the data reception unit 97 of the control device 90 receives the detection data of the distance detection device 20 via the communication system 300 (step S210).
- the data receiving unit 97 of the control device 90 receives detection data of the position detection device 30 via the communication system 300 (step S220).
- the data receiving unit 97 of the control device 90 receives the detection data of the posture detection device 40 via the communication system 300 (step S230).
- the data receiving unit 97 of the control device 90 receives the image data of the imaging device 50 via the communication system 300 (step S240).
- the data receiving unit 97 of the control device 90 receives the detection data of the position detection device 430 via the communication system 300 (step S250).
- the data receiving unit 97 of the control device 90 receives the detection data of the posture detection device 440 via the communication system 300 (step S260).
- step S210, step S220, step S230, step S240, step S250, and step S260 are performed is arbitrary, and may be performed simultaneously.
- the target image generation unit 91 generates a target image AGa indicating a virtual viewpoint image of the target WAa based on the detection data of the distance detection device 20 (step S270).
- the working machine image generation unit 92 generates a working machine image BG indicating a virtual viewpoint image of the hydraulic shovel 1 based on the detection data of the position detection device 30 and the detection data of the posture detection device 40 (step S280).
- Transport vehicle image generation unit 95 generates transport vehicle image AGb showing a virtual viewpoint image of transport vehicle WAb based on the detection data of position detection device 430 and the detection data of posture detection device 440 (step S290).
- the combining unit 96 combines the target image AGa, the work machine image BG, and the transport vehicle image AGb to generate an imposed image IG (step S300).
- the combining unit 96 generates an imposed image IG in which the target image AGa, the work machine image BG, and the transport vehicle image AGb are superimposed.
- the combining unit 96 uses the coordinate conversion data stored in the coordinate conversion data storage unit 94 to display the target image AGa, the work machine image BG, and the transport vehicle image AGb in a single coordinate system.
- the coordinate system of the target image AGa, the coordinate system of the work machine image BG, and the coordinate system of the transport vehicle image AGb are adjusted.
- the display control unit 93 outputs the actual image RG of the target WAa captured by the imaging device 50 to the display device 80. In addition, the display control unit 93 acquires the imposed image IG generated by the combining unit 96 from the combining unit 96. The display control unit 93 outputs an imposed image IG including the target image AGa, the work machine image BG, and the transport vehicle image AGb defined in a single coordinate system to the display device 80 (step S310).
- the display device 80 displays the real image RG of the object WAa including at least the real image of the work implement 4 and the pose image IG (step S320).
- the display control unit 93 causes the display screen of the display device 80 to display the actual image of the work implement 4 and the imposed image IG at different positions.
- the operator operates the remote control device 70 to remotely control the hydraulic shovel 1 while viewing the actual image RG and the pose image IG displayed on the display device 80.
- the imposed image IG is a three-dimensional image looking over the object WAa, the transport vehicle WAb, and the hydraulic shovel 1
- the operator senses the perspective of the hydraulic shovel 1 and the object WAa by looking at the imposed image IG.
- the target image AGa showing the three-dimensional shape of the ground surface which is the target WAa is displayed
- the operator looks at the target image AGa and the work machine image BG and The distance to the work implement 4 can be grasped. Therefore, the operator can smoothly carry out the digging operation or the leveling operation.
- the operator looks at the transport vehicle image AGb and the work machine image BG, and the work machine 4 of the transport vehicle WAb and the hydraulic shovel 1 You can grasp the distance with. Therefore, the operator can smoothly carry out the loading operation and the like.
- FIG. 15 is a view schematically showing an example of the impose image IG according to the present embodiment.
- FIG. 15 shows an impose image IG including a work machine image BG and a transport vehicle image AGb defined in the vehicle body coordinate system.
- the display control unit 93 causes the display device 80 to display the work machine image BG and the transport vehicle image AGb viewed from the virtual viewpoint fixed in the vehicle body coordinate system.
- the virtual viewpoint is fixed in the vehicle coordinate system. Therefore, when the swing body 3 turns in a state where the traveling body 2 of the hydraulic shovel 1 is stopped, the swinging body image 3G of the working machine image BG is displayed as being stopped, and the traveling body image 2G is the turning axis Zr It is displayed to turn around. Further, the transport vehicle image AGb is also displayed so as to turn about the turning axis Zr.
- FIG. 16 is a view schematically showing an example of the impose image IG according to the present embodiment.
- FIG. 16 shows an imposed image IG including a work machine image BG and a transport vehicle image AGb defined in the global coordinate system.
- the display control unit 93 causes the display device 80 to display the work machine image BG and the transport vehicle image AGb viewed from the virtual viewpoint fixed in the global coordinate system.
- the virtual viewpoint is fixed in the global coordinate system. Therefore, when the swing body 3 turns in a state where the traveling body 2 of the hydraulic shovel 1 is stopped, the traveling body image 2G in the working machine image BG is displayed as being stopped, and the swing body image 3G is the turning axis Zr It is displayed to turn around. In addition, when the transport vehicle WAb is stopped, the transport vehicle image AGb is displayed as being stopped.
- the working machine image BG seen from the virtual viewpoint outside the hydraulic shovel 1 is generated based on the detection data of the position detection device 30 and the detection data of the posture detection device 40. Ru. Further, based on the detection data of the position detection device 430 and the detection data of the posture detection device 440, a transport vehicle image AGb seen from a virtual viewpoint outside the hydraulic shovel 1 and the transport vehicle WAb is generated.
- the work machine image BG and the transport vehicle image AGb are three-dimensional computer graphics, and the pose image IG including the work machine image BG and the transport vehicle image AGb is displayed on the display device 80.
- the operator can obtain a sense of perspective of the work site by viewing the work machine image BG and the transport vehicle image AGb displayed on the display device 80. Therefore, the operator can easily grasp the distance between the transport vehicle WAb and the hydraulic excavator 1, particularly the distance between the loading platform 401 of the transport vehicle WAb and the bucket 8 of the hydraulic excavator 1. Therefore, the operator can operate the remote control device 70 and use the work implement 4 of the hydraulic shovel 1 to smoothly carry out the loading operation on the transport vehicle WAb. As a result, a reduction in the remote control operation efficiency is suppressed.
- the impose image IG is an image in which the work machine image BG and the transport vehicle image AGb are superimposed. Further, in the present embodiment, when the traveling body 2, the revolving body 3 and the working machine 4 move, the traveling body image 2 G, the revolving body image 3 G and the working machine image 4 G are synchronized in the pose image IG. When the moving vehicle WAb moves, the moving vehicle image AGb moves in synchronization in the imposed image IG.
- the traveling body image 2G, the revolving body image 3G and the working machine image 4G At least one moves in a state of being superimposed on the transport vehicle image AGb. Therefore, by looking at the imposed image IG in which the working machine image BG and the transporting vehicle image AGb are superimposed, the operator carries with at least one of the traveling body 2, the rotating body 3 and the working machine 4 in the moving state. It becomes easy to recognize the sense of perspective with the vehicle WAb.
- the present embodiment not only the work machine image BG and the transport vehicle image AGb, but also the target image AGa of the target WAa of the work site including the construction target is displayed in the impose image IG.
- the operator can easily grasp the distance between the object WAa and the hydraulic excavator 1. Therefore, the operator can operate the remote control device 70 to smoothly construct the target WAa with the work machine 4 of the hydraulic shovel 1.
- the distance detection device 20, the position detection device 30, the posture detection device 40, and the imaging device 50 are mounted on the hydraulic shovel 1.
- the distance detection device is installed outside the hydraulic shovel 1, it is necessary to separately prepare the distance detection device or to carry out the installation work of the distance detection device. In this case, the device cost may increase or the work efficiency may decrease.
- the distance detection device 20 By mounting the distance detection device 20 on the hydraulic shovel 1, it is not necessary to separately prepare the distance detection device, and it is not necessary to carry out the work of installing the distance detection device. Therefore, the device cost can be suppressed, and a decrease in work efficiency can be suppressed.
- the distance detection device 20 by mounting the distance detection device 20 on the hydraulic shovel 1, even when the position of the target WAa (construction target) gradually changes, it is not necessary to carry out the work of installing the distance detection device. The same applies to the position detection device 30, the posture detection device 40, and the imaging device 50.
- the pose image IG including the work machine image BG and the transport vehicle image AGb is displayed on a part of the display screen of the display device 80 including the actual image RG captured by the imaging device 50.
- the operator can view the pose image IG without significantly moving the line of sight while watching the real image RG. Therefore, the operator can feel a sense of security and can perform smooth operation.
- the actual image RG to which the impose image IG is imposed includes the image of the work implement 4. That is, the imposed image IG is imposed on the real image RG in the + Xm direction of the revolving unit 3.
- the operator can grasp the work situation of the working machine 4 by looking at the actual image RG, and can grasp the distance between the transport vehicle WAb and the work machine 4 by looking at the imposed image IG. .
- the operator operates the remote control device 70 while looking at the actual image of the work machine 4.
- the operator can pose the pose image IG without significantly moving the line of sight while looking at the real image RG. You can see Therefore, since the operator can carry out smooth operation, a reduction in work efficiency is suppressed. Further, on the display screen of the display device 80, the actual image of the work implement 4 and the imposed image IG are displayed at different positions. As a result, since the real image of the work implement 4 is not blocked by the pose image IG, the operator can smoothly view both the real image of the work implement 4 and the pose image IG.
- the work machine image BG and the transport vehicle image AGb are displayed in a predetermined single coordinate system.
- the operator can easily grasp the distance and relative position between the hydraulic shovel 1 and the transport vehicle image AGb by viewing the work machine image BG and the transport vehicle image AGb.
- the work machine image BG and the transport vehicle image AGb are generated in the vehicle body coordinate system, and the work machine image BG and the transport vehicle image AGb viewed from the virtual viewpoint fixed in the vehicle body coordinate system are displayed.
- the swing body image 3G is stopped, and the traveling body image 2G and the transport vehicle image AGb are displayed so as to turn. Thereby, the operator can look at the situation of the work site based on the revolving unit 3.
- the work machine image BG and the transport vehicle image AGb are generated in the global coordinate system, and the work machine image BG and the transport vehicle image AGb seen from the virtual viewpoint fixed in the global coordinate system are displayed.
- the traveling body image 2G and the transport vehicle image AGb are stopped, and the swing body image 3G is displayed so as to turn. Thereby, the operator can look at the situation of the work site based on the traveling body 2 and the transport vehicle image AGb.
- the work machine image BG includes not only the image of the work machine 4 but also the image of the revolving unit 3. Therefore, the operator looks at the target image AGa displayed on the display device 80, and not only the operation status of the work implement 4 and the distance between the target WAa and the work implement 4 but also the turning status of the revolving unit 3 and the target WAa The relative position with respect to the revolving unit 3 can be grasped. Further, by providing a swing body posture sensor 41 for detecting the posture of the swing body 3 and a work machine posture sensor 42 for detecting the posture of the work machine 4, actual data can be obtained based on detection data of the swing body posture sensor 41.
- a swinging body image 3G is displayed so as to be interlocked with the swinging of the swinging body 3, and a work implement image 4G is displayed so as to be interlocked with the actual operation of the work implement 4 based on detection data of the work implement posture sensor 42 .
- the operator can check the posture and operating condition of the hydraulic shovel 1.
- the posture detection device 440 including the IMU is mounted on the transport vehicle WAb, and the posture detection device 440 detects the posture (roll angle ⁇ 4, pitch angle ⁇ 5, yaw angle ⁇ 6) of the transport vehicle WAb. It was decided.
- the yaw angle ⁇ 6 may be detected as the posture of the transport vehicle WAb, and the roll angle ⁇ 4 and the pitch angle ⁇ 5 may not be detected.
- the transport vehicle WAb is often positioned on the ground substantially parallel to the horizontal plane, and the roll angle ⁇ 4 and the pitch angle ⁇ 5 are often close to zero. Therefore, the transport vehicle image AGb may be displayed based on the detection data of the yaw angle ⁇ 6 in a state where the roll angle ⁇ 4 and the pitch angle ⁇ 5 are regarded as zero.
- the posture detection device 440 may be omitted.
- the yaw angle ⁇ 6 can be detected by the position detection device 430.
- detection data of the position detection device 430 is output to the control device 460.
- the arithmetic processing unit 460A of the control unit 460 includes an attitude calculation unit 440B that calculates the yaw angle ⁇ 6 based on the detection data of the position detection unit 430.
- the posture calculation unit 440B determines the position of one GPS antenna 431 and the other GPS antenna.
- Posture calculation unit 440B Based on the 431 position, the orientation of the vehicle body 402 with respect to the reference orientation in the global coordinate system can be detected.
- Posture calculation unit 440B performs arithmetic processing based on the position of one GPS antenna 431 and the position of the other GPS antenna 431, and detects the direction of vehicle body 402 with respect to the reference direction.
- the angle between the reference orientation and the orientation of the vehicle body 402 corresponds to the yaw angle ⁇ 6.
- the attitude calculation unit 440B calculates a straight line connecting the position of one GPS antenna 431 and the position of the other GPS antenna 431, and detects the yaw angle ⁇ 6 based on the angle formed by the calculated straight line and the reference direction. Can.
- the number of GPS antennas 431 (position detection device 430) provided in the transport vehicle WAb may be one.
- Posture calculation unit 440 ⁇ / b> B can derive a movement trajectory of transport vehicle WAb in the horizontal plane based on detection data of one position detection device 430.
- the direction of the tangent of the movement trajectory of the transport vehicle WAb with respect to the reference orientation corresponds to the yaw angle ⁇ 6.
- the attitude calculation unit 440B can detect the yaw angle ⁇ 6 based on the tangent of the movement trajectory derived based on the detection data of one position detection device 430.
- the three-dimensional shape of the target image AGa and the image captured by the imaging device 50 may be displayed in the superimposed image IG in a superimposed state.
- the three-dimensional shape of the object WAa in the imaging device coordinate system by converting the position Pd in the three-dimensional shape of the object WAa detected by the distance detection device 20 into the position Pc in the image of the imaging device coordinate system. And the image of the object WAa can be superimposed. Further, the position Pd in the three-dimensional shape of the object WAa and the position Pc in the image of the object WAa can be converted into the position Pm of the vehicle coordinate system or the position Pg of the global coordinate system.
- the impose image IG includes the work machine image BG, the transport vehicle image AGb, and the target image AGa.
- the impose image IG includes the work machine image BG and the transport vehicle image AGb, and may not include the target image AGa.
- the distance detection device 20 may be omitted.
- the load image IG is loaded, for example It is possible to suppress the occurrence of an unexpected situation in which the loading platform 401 and the bucket 8 are brought into contact in the operation.
- the work machine image BG includes the traveling body image 2G, the revolving body image 3G, and the work machine image 4G.
- the work machine image BG may include the swing body image 3G and the work machine image 4G and may not include the traveling body image 2G.
- the work machine image BG may include the work machine image 4G and may not include the traveling body image 2G and the revolving body image 3G.
- the distance detection device 20 is a laser range finder.
- the distance detection device 20 is not limited to the laser range finder.
- the distance detection device 20 may be, for example, an ultrasonic distance sensor or a stereo camera as long as it can detect the three-dimensional shape of the object WAa.
- a plurality of distance detection devices 20 may be provided. Further, by the distance detection device 20, not only the three-dimensional shape of the object WAa in the forward direction of the swing body 3 but also the three-dimensional shape of the object WAa in the right direction of the swing body 3 and the object WAa in the left direction of the swing body 3 At least one of the three-dimensional shape and the three-dimensional shape of the object WAa in the backward direction of the swing body 3 may be detected.
- the work machine attitude sensor 42 is a stroke sensor.
- Work implement attitude sensor 42 is not limited to a stroke sensor.
- the work machine attitude sensor 42 may be, for example, a rotary sensor, an acceleration sensor, or an optical sensor such as a laser scanner as long as it can detect the tilt angle of the work machine 4.
- the imaging device 50 is a monocular camera.
- the imaging device 50 is not limited to a monocular camera.
- the imaging device 50 may be, for example, a stereo camera having a function of detecting a three-dimensional shape of the object WAa and a function of acquiring an image.
- a plurality of imaging devices 50 may be provided. Further, by the imaging device 50, not only the image of the target WAa in the forward direction of the swing body 3 but also the image of the target WAa in the right direction of the swing body 3, the image of the target WAa in the left direction of the swing body 3, and the swing body 3 At least one of the images of the backward target WAa may be acquired.
- At least one of the distance detection device 20, the position detection device 30, the posture detection device 40, and the imaging device 50 is installed outside the hydraulic shovel 1 without being mounted on the hydraulic shovel 1. It is also good.
- the position Pb of the transport vehicle WAb is detected using GNSS.
- the position of the transport vehicle WAb may be detected based on, for example, a stereo camera or a laser range finder mounted on the hydraulic excavator 1, or detected by a GNSS function such as a smartphone possessed by the driver of the transport vehicle WAb. It is also good.
- At least one of the position detection device 430 and the posture detection device 440 may be installed outside the transport vehicle WAb without being mounted on the transport vehicle WAb.
- the virtual viewpoint may move. That is, the display control unit 93 may cause the display device 80 to display the pose image IG including the work machine image BG and the transport vehicle image AGb as viewed from the virtual viewpoints set at each of a plurality of different positions.
- the pose image IG may be a free viewpoint image.
- the free viewpoint image refers to an image viewed from the set virtual viewpoint with the virtual viewpoint for the object (hydraulic shovel 1 and transport vehicle 2) set at an arbitrary position.
- the detection data of the distance detection device 20, the detection data of the position detection device 30, and the detection data of the attitude detection device 40 are transmitted to the control device 90 through the communication system 300.
- the work machine image BG and the transport vehicle image AGb are generated and displayed on the display device 80.
- At least one of the working machine image BG and the transporting vehicle image AGb is generated by the control device 60 mounted on the hydraulic shovel 1, and at least one of the generated working machine image BG and the transporting vehicle image AGb is the communication system 300. It may be transmitted to the control device 90 via the display device 80 and displayed on the display device 80.
- the detection data of the position detection device 430 and the detection data of the attitude detection device 440 are transmitted to the control device 90 through the communication system 300, and the control device 90 generates the transport vehicle image AGb. , And displayed on the display device 80.
- the transport vehicle image AGb is generated by the control device 460 mounted on the transport vehicle WAb, and the generated transport vehicle image AGb is transmitted to the control device 90 via the communication system 300 and displayed on the display device 80. It is also good.
- the pose image IG is displayed on part of the display screen of the display device 80 that displays the actual image RG of the work site.
- the display device that displays the real image RG and the display device that displays the imposed image IG may be different display devices.
- the sub display may be disposed adjacent to the main display, the real image RG may be displayed on the main display, and the pose image IG may be displayed on the sub display.
- the working machine 1 is a hydraulic shovel.
- the work machine 1 only needs to have a work machine and a swing body that supports the work machine, and is not limited to a hydraulic shovel.
- the work machine 1 only needs to have a work machine, and may not have a swing body.
- the work machine 1 may be a wheel loader or the like.
- SYMBOLS 1 Hydraulic shovel (work machine), 2 ... Traveling body, 2A ... Track, 2B ... Track, 3 ... Swirl body, 4 ... Work machine, 5 ... Hydraulic cylinder, 6 ... Boom, 7 ... Arm, 8 ... Bucket, 9 ...
- tip end portion 10 boom cylinder 11: arm cylinder 12: bucket cylinder 20: distance detection device 30: position detection device (first position detection device) 31: GPS antenna 40: posture detection device 1 Posture detection device) 41: Swing body posture sensor 42: Work machine posture sensor 42A: Boom stroke sensor 42B: Arm stroke sensor 42C: Bucket stroke sensor 50: Imaging device 60: Control device 60A: 60A Arithmetic processing unit, 60B: storage unit, 60C: input / output interface, 70: remote control unit, 71: working lever, 71L: left working lever, 71R Right working lever, 72: running lever, 72L: left running lever, 72R: right running lever, 80: display device, 90: control device, 90A: arithmetic processing unit, 90B: storage device, 90C: input / output interface, 91: Target image generation unit 92: Work machine image generation unit 93: Display control unit 94: Coordinate conversion data storage unit 95: Transport vehicle image generation unit 96: Synthesis unit 97: Data reception unit 100: Remote control System 200 display
Landscapes
- Engineering & Computer Science (AREA)
- Mining & Mineral Resources (AREA)
- Civil Engineering (AREA)
- General Engineering & Computer Science (AREA)
- Structural Engineering (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Component Parts Of Construction Machinery (AREA)
- Operation Control Of Excavators (AREA)
- Selective Calling Equipment (AREA)
- Telephonic Communication Services (AREA)
Abstract
Description
図1は、本実施形態に係る作業機械1の遠隔操作システム100の一例を模式的に示す図である。遠隔操作システム100は、作業機械1を遠隔操作する。本実施形態においては、作業機械1が油圧ショベルであることとする。以下の説明においては、作業機械1を適宜、油圧ショベル1、と称する。
図2は、本実施形態に係る油圧ショベル1を模式的に示す側面図である。図3は、本実施形態に係る油圧ショベル1を模式的に示す背面図である。図4は、本実施形態に係る油圧ショベル1を模式的に示す平面図である。
図5は、本実施形態に係る運搬車両WAbを模式的に示す側面図である。図6は、本実施形態に係る運搬車両WAbを模式的に示す背面図である。図7は、本実施形態に係る運搬車両WAbを模式的に示す平面図である。
図8は、本実施形態に係る距離検出装置20及び撮像装置50を模式的に示す図である。上述のように、本実施形態においては、グローバル座標系(Xg-Yg-Zg座標系)と、車体座標系(Xm-Ym-Zm座標系)とが規定される。
図9は、本実施形態に係る距離検出装置20及び撮像装置50の一例を模式的に示す図である。図9は、距離検出装置20によって検出され撮像装置50によって撮像される対象WAaが施工対象の地表である例を示す。
図10は、本実施形態に係る制御装置60、制御装置460、及び制御装置90の一例を示す機能ブロック図である。
図12、図13、及び図14は、本実施形態に係る表示方法の一例を示すフローチャートである。図12は、油圧ショベル1の制御装置60の処理を示すフローチャートである。図13は、運搬車両WAbの制御装置460の処理を示すフローチャートである。図14は、遠隔操作システム100の制御装置90の処理を示すフローチャートである。図12、図13、及び図14に示す処理は、所定の周期で実施される。
以上説明したように、本実施形態によれば、位置検出装置30の検出データ及び姿勢検出装置40の検出データに基づいて、油圧ショベル1の外部の仮想視点から見た作業機械画像BGが生成される。また、位置検出装置430の検出データ及び姿勢検出装置440の検出データに基づいて、油圧ショベル1及び運搬車両WAbの外部の仮想視点から見た運搬車両画像AGbが生成される。作業機械画像BG及び運搬車両画像AGbは、3次元コンピュータグラフィックスであり、それら作業機械画像BG及び運搬車両画像AGbを含むインポーズ画像IGが表示装置80に表示される。操作者は、表示装置80に表示された作業機械画像BG及び運搬車両画像AGbを見ることにより、作業現場の遠近感を得ることができる。そのため、操作者は、運搬車両WAbと油圧ショベル1との距離、特に運搬車両WAbの荷台401と油圧ショベル1のバケット8との距離を容易に把握することができる。したがって、操作者は、遠隔操作装置70を操作して、油圧ショベル1の作業機4を使って、運搬車両WAbに対する積込作業を円滑に実施することができる。これにより、遠隔操作の作業効率の低下が抑制される。
Claims (9)
- 作業機を有する作業機械の位置の検出データと前記作業機械の姿勢の検出データとに基づいて、前記作業機械の外部の仮想視点から見た前記作業機械の仮想視点画像を示す作業機械画像を生成する作業機械画像生成部と、
運搬車両の位置の検出データと前記運搬車両の姿勢の検出データとに基づいて、前記仮想視点から見た前記運搬車両の仮想視点画像を示す運搬車両画像を生成する運搬車両画像生成部と、
前記作業機械画像と前記運搬車両画像とを重畳した合成画像を生成する合成部と、
前記合成画像を前記作業機械の外部に存在する表示装置に表示させる表示制御部と、
を備える表示システム。 - 前記表示制御部は、前記作業機械に搭載された撮像装置で撮影された前記作業機の実画像と前記合成画像とを前記表示装置に同時に表示させる、
請求項1に記載の表示システム。 - 前記表示制御部は、前記表示装置の表示画面において前記作業機の実画像と前記合成画像とを異なる位置に表示させる、
請求項2に記載の表示システム。 - 前記表示制御部は、所定の座標系において前記合成画像を表示させる、
請求項1から請求項3のいずれか一項に記載の表示システム。 - 前記座標系は、前記作業機を支持する旋回体に規定された車体座標系であり、
前記表示制御部は、前記車体座標系において固定された前記仮想視点から見た前記合成画像を表示させる、
請求項4に記載の表示システム。 - 前記座標系は、グローバル座標系であり、
前記表示制御部は、前記グローバル座標系において固定された前記仮想視点から見た前記合成画像を表示させる、
請求項4に記載の表示システム。 - 前記作業機械の姿勢は、前記作業機械に搭載された第1姿勢検出装置より検出され、
前記第1姿勢検出装置は、前記作業機の姿勢を検出する作業機姿勢センサと前記作業機を支持する旋回体の姿勢を検出する旋回体姿勢センサとを含み、
前記作業機械画像は、前記作業機の画像と前記旋回体の画像とを含む、
請求項1から請求項6のいずれか一項に記載の表示システム。 - 制御装置が、
作業機を有する作業機械の位置の検出データと前記作業機械の姿勢の検出データとを受信し、
運搬車両の位置の検出データと前記運搬車両の姿勢の検出データとを受信し、
前記作業機械の位置の検出データと前記作業機械の姿勢の検出データとに基づいて、前記作業機械の外部の仮想視点から見た前記作業機械の仮想視点画像を示す作業機械画像を生成し、
前記運搬車両の位置の検出データと前記運搬車両の姿勢の検出データとに基づいて、前記仮想視点から見た前記運搬車両の仮想視点画像を示す運搬車両画像を生成し、
前記作業機械画像と前記運搬車両画像とを重畳した合成画像を生成し、
前記合成画像を前記作業機械の外部に存在する表示装置に表示させる、
表示方法。 - 作業機を有する作業機械の外部の仮想視点から見た前記作業機械の仮想視点画像を示す作業機械画像と前記仮想視点から見た運搬車両の仮想視点画像を示す運搬車両画像とを重畳した合成画像を表示する表示装置。
Priority Applications (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
AU2018333191A AU2018333191B2 (en) | 2017-09-15 | 2018-06-25 | Display system, display method, and display apparatus |
CA3054897A CA3054897A1 (en) | 2017-09-15 | 2018-06-25 | Display system, display method, and display apparatus |
US16/491,002 US11230825B2 (en) | 2017-09-15 | 2018-06-25 | Display system, display method, and display apparatus |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2017178316A JP6898816B2 (ja) | 2017-09-15 | 2017-09-15 | 表示システム、表示方法、及び表示装置 |
JP2017-178316 | 2017-09-15 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2019054000A1 true WO2019054000A1 (ja) | 2019-03-21 |
Family
ID=65723570
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/JP2018/024065 WO2019054000A1 (ja) | 2017-09-15 | 2018-06-25 | 表示システム、表示方法、及び表示装置 |
Country Status (5)
Country | Link |
---|---|
US (1) | US11230825B2 (ja) |
JP (1) | JP6898816B2 (ja) |
AU (1) | AU2018333191B2 (ja) |
CA (1) | CA3054897A1 (ja) |
WO (1) | WO2019054000A1 (ja) |
Cited By (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20200370282A1 (en) * | 2016-01-29 | 2020-11-26 | Sumitomo(S.H.I.) Construction Machinery Co., Ltd. | Shovel and autonomous aerial vehicle flying around shovel |
WO2021020292A1 (ja) * | 2019-07-26 | 2021-02-04 | 株式会社小松製作所 | 表示システム、遠隔操作システム、及び表示方法 |
JP7349947B2 (ja) | 2020-03-26 | 2023-09-25 | 住友重機械工業株式会社 | 情報処理装置、作業機械、情報処理方法、情報処理プログラム |
Families Citing this family (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
FI20176052A1 (en) * | 2017-11-24 | 2019-05-25 | Novatron Oy | Control of earthmoving machinery |
JP7155516B2 (ja) * | 2017-12-20 | 2022-10-19 | コベルコ建機株式会社 | 建設機械 |
JP7199864B2 (ja) * | 2018-07-31 | 2023-01-06 | 株式会社小松製作所 | 作業機械を制御するためのシステム及び方法 |
JP7041320B2 (ja) * | 2019-03-19 | 2022-03-23 | 日立建機株式会社 | ダンプトラックの荷台昇降装置 |
JP7306191B2 (ja) * | 2019-09-26 | 2023-07-11 | コベルコ建機株式会社 | 輸送車位置判定装置 |
JP2021136462A (ja) * | 2020-02-21 | 2021-09-13 | 株式会社小松製作所 | 作業機械の遠隔操作システム |
AU2022261652A1 (en) * | 2021-04-19 | 2023-10-26 | Orica International Pte Ltd | Fragmentation georeferencing |
CN113737886B (zh) * | 2021-09-26 | 2023-02-17 | 广西柳工机械股份有限公司 | 遥控装载机卸料辅助系统和方法 |
Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2005071619A1 (ja) * | 2004-01-21 | 2005-08-04 | Campus Create Co., Ltd. | 画像生成方法 |
WO2014123228A1 (ja) * | 2013-02-08 | 2014-08-14 | 日立建機株式会社 | 旋回式作業機械の周囲監視装置 |
JP2015021246A (ja) * | 2013-07-17 | 2015-02-02 | 日立建機株式会社 | 作業機械の周囲監視装置 |
Family Cites Families (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2013168776A (ja) | 2012-02-15 | 2013-08-29 | Fujita Corp | 遠隔操作システム |
JP5996879B2 (ja) * | 2012-02-16 | 2016-09-21 | 日立建機株式会社 | 作業機械の周囲監視装置 |
US8965642B2 (en) | 2012-10-05 | 2015-02-24 | Komatsu Ltd. | Display system of excavating machine and excavating machine |
CA2923683C (en) | 2013-12-12 | 2018-03-13 | Hitachi Construction Machinery Co., Ltd. | Vehicle travel system and vehicle travel control method |
JP2016065769A (ja) * | 2014-09-24 | 2016-04-28 | 日立建機株式会社 | 運搬車両 |
JP6407663B2 (ja) * | 2014-10-30 | 2018-10-17 | 日立建機株式会社 | 作業支援画像生成装置、及びそれを備えた作業機械の操縦システム |
JP6777375B2 (ja) | 2015-03-05 | 2020-10-28 | 株式会社小松製作所 | 作業機械の画像表示システム、作業機械の遠隔操作システム及び作業機械 |
-
2017
- 2017-09-15 JP JP2017178316A patent/JP6898816B2/ja active Active
-
2018
- 2018-06-25 WO PCT/JP2018/024065 patent/WO2019054000A1/ja active Application Filing
- 2018-06-25 CA CA3054897A patent/CA3054897A1/en not_active Withdrawn
- 2018-06-25 AU AU2018333191A patent/AU2018333191B2/en active Active
- 2018-06-25 US US16/491,002 patent/US11230825B2/en active Active
Patent Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2005071619A1 (ja) * | 2004-01-21 | 2005-08-04 | Campus Create Co., Ltd. | 画像生成方法 |
WO2014123228A1 (ja) * | 2013-02-08 | 2014-08-14 | 日立建機株式会社 | 旋回式作業機械の周囲監視装置 |
JP2015021246A (ja) * | 2013-07-17 | 2015-02-02 | 日立建機株式会社 | 作業機械の周囲監視装置 |
Cited By (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20200370282A1 (en) * | 2016-01-29 | 2020-11-26 | Sumitomo(S.H.I.) Construction Machinery Co., Ltd. | Shovel and autonomous aerial vehicle flying around shovel |
US11492783B2 (en) * | 2016-01-29 | 2022-11-08 | Sumitomo(S.H.I) Construction Machinery Co., Ltd. | Shovel and autonomous aerial vehicle flying around shovel |
WO2021020292A1 (ja) * | 2019-07-26 | 2021-02-04 | 株式会社小松製作所 | 表示システム、遠隔操作システム、及び表示方法 |
JP7351478B2 (ja) | 2019-07-26 | 2023-09-27 | 株式会社小松製作所 | 表示システム、遠隔操作システム、及び表示方法 |
US11939744B2 (en) | 2019-07-26 | 2024-03-26 | Komatsu Ltd. | Display system, remote operation system, and display method |
JP7349947B2 (ja) | 2020-03-26 | 2023-09-25 | 住友重機械工業株式会社 | 情報処理装置、作業機械、情報処理方法、情報処理プログラム |
Also Published As
Publication number | Publication date |
---|---|
AU2018333191B2 (en) | 2021-07-29 |
JP2019054465A (ja) | 2019-04-04 |
JP6898816B2 (ja) | 2021-07-07 |
CA3054897A1 (en) | 2019-03-21 |
US11230825B2 (en) | 2022-01-25 |
AU2018333191A1 (en) | 2019-09-26 |
US20200018049A1 (en) | 2020-01-16 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
JP6898816B2 (ja) | 表示システム、表示方法、及び表示装置 | |
JP6927821B2 (ja) | 表示システム、及び表示装置 | |
JP6777375B2 (ja) | 作業機械の画像表示システム、作業機械の遠隔操作システム及び作業機械 | |
JP6925816B2 (ja) | 位置計測システム、作業機械、及び位置計測方法 | |
JP2020197431A (ja) | マップ生成システム及びマップ生成方法 | |
KR102248026B1 (ko) | 작업 기계의 계측 시스템, 작업 기계, 및 작업 기계의 계측 방법 | |
JP2022164713A (ja) | 作業機械の画像表示システム及び作業機械の画像表示方法 | |
AU2020320149B2 (en) | Display system, remote operation system, and display method | |
JP2021165526A (ja) | 作業機械の画像表示システム | |
JP7333551B2 (ja) | 作業機械の画像表示システム | |
US20210395980A1 (en) | System and method for work machine | |
JP2021011814A (ja) | 作業機械の画像表示システム | |
JP2020197045A (ja) | 表示システムおよび表示方法 | |
JP2021022827A (ja) | 表示システム、遠隔操作システム、及び表示方法 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 18855959 Country of ref document: EP Kind code of ref document: A1 |
|
ENP | Entry into the national phase |
Ref document number: 3054897 Country of ref document: CA |
|
ENP | Entry into the national phase |
Ref document number: 2018333191 Country of ref document: AU Date of ref document: 20180625 Kind code of ref document: A |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 18855959 Country of ref document: EP Kind code of ref document: A1 |