WO2018043299A1 - 作業機械の画像表示システム、作業機械の遠隔操作システム、作業機械及び作業機械の画像表示方法 - Google Patents
作業機械の画像表示システム、作業機械の遠隔操作システム、作業機械及び作業機械の画像表示方法 Download PDFInfo
- Publication number
- WO2018043299A1 WO2018043299A1 PCT/JP2017/030406 JP2017030406W WO2018043299A1 WO 2018043299 A1 WO2018043299 A1 WO 2018043299A1 JP 2017030406 W JP2017030406 W JP 2017030406W WO 2018043299 A1 WO2018043299 A1 WO 2018043299A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- work machine
- route
- image
- work
- display
- Prior art date
Links
- 238000000034 method Methods 0.000 title claims description 17
- 238000012545 processing Methods 0.000 claims abstract description 92
- 238000004364 calculation method Methods 0.000 claims abstract description 82
- 238000003384 imaging method Methods 0.000 claims description 67
- 238000001514 detection method Methods 0.000 description 30
- 238000006243 chemical reaction Methods 0.000 description 24
- 238000004891 communication Methods 0.000 description 22
- 239000011159 matrix material Substances 0.000 description 19
- 238000010586 diagram Methods 0.000 description 15
- 238000012986 modification Methods 0.000 description 11
- 230000004048 modification Effects 0.000 description 11
- 230000009467 reduction Effects 0.000 description 6
- 230000006870 function Effects 0.000 description 5
- 230000008569 process Effects 0.000 description 5
- 238000002485 combustion reaction Methods 0.000 description 4
- 239000000203 mixture Substances 0.000 description 4
- 238000010248 power generation Methods 0.000 description 4
- 230000009466 transformation Effects 0.000 description 4
- 238000013461 design Methods 0.000 description 3
- 238000006073 displacement reaction Methods 0.000 description 3
- 239000010720 hydraulic oil Substances 0.000 description 3
- 208000034530 PLAA-associated neurodevelopmental disease Diseases 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 238000011960 computer-aided design Methods 0.000 description 2
- 238000005553 drilling Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 239000011435 rock Substances 0.000 description 2
- 239000002689 soil Substances 0.000 description 2
- 240000004050 Pentaglottis sempervirens Species 0.000 description 1
- 235000004522 Pentaglottis sempervirens Nutrition 0.000 description 1
- 230000001133 acceleration Effects 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 230000000295 complement effect Effects 0.000 description 1
- 239000002131 composite material Substances 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000007599 discharging Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000005484 gravity Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 229910044991 metal oxide Inorganic materials 0.000 description 1
- 150000004706 metal oxides Chemical class 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Images
Classifications
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60K—ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
- B60K35/00—Arrangement of adaptations of instruments
-
- E—FIXED CONSTRUCTIONS
- E02—HYDRAULIC ENGINEERING; FOUNDATIONS; SOIL SHIFTING
- E02F—DREDGING; SOIL-SHIFTING
- E02F9/00—Component parts of dredgers or soil-shifting machines, not restricted to one of the kinds covered by groups E02F3/00 - E02F7/00
- E02F9/16—Cabins, platforms, or the like, for drivers
-
- B60K35/23—
-
- B60K35/28—
-
- B60K35/60—
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B62—LAND VEHICLES FOR TRAVELLING OTHERWISE THAN ON RAILS
- B62D—MOTOR VEHICLES; TRAILERS
- B62D1/00—Steering controls, i.e. means for initiating a change of direction of the vehicle
- B62D1/24—Steering controls, i.e. means for initiating a change of direction of the vehicle not vehicle-mounted
- B62D1/28—Steering controls, i.e. means for initiating a change of direction of the vehicle not vehicle-mounted non-mechanical, e.g. following a line or other known markers
-
- E—FIXED CONSTRUCTIONS
- E02—HYDRAULIC ENGINEERING; FOUNDATIONS; SOIL SHIFTING
- E02F—DREDGING; SOIL-SHIFTING
- E02F9/00—Component parts of dredgers or soil-shifting machines, not restricted to one of the kinds covered by groups E02F3/00 - E02F7/00
- E02F9/20—Drives; Control devices
- E02F9/2025—Particular purposes of control systems not otherwise provided for
- E02F9/205—Remotely operated machines, e.g. unmanned vehicles
-
- E—FIXED CONSTRUCTIONS
- E02—HYDRAULIC ENGINEERING; FOUNDATIONS; SOIL SHIFTING
- E02F—DREDGING; SOIL-SHIFTING
- E02F9/00—Component parts of dredgers or soil-shifting machines, not restricted to one of the kinds covered by groups E02F3/00 - E02F7/00
- E02F9/26—Indicating devices
- E02F9/261—Surveying the work-site to be treated
- E02F9/262—Surveying the work-site to be treated with follow-up actions to control the work tool, e.g. controller
-
- E—FIXED CONSTRUCTIONS
- E02—HYDRAULIC ENGINEERING; FOUNDATIONS; SOIL SHIFTING
- E02F—DREDGING; SOIL-SHIFTING
- E02F9/00—Component parts of dredgers or soil-shifting machines, not restricted to one of the kinds covered by groups E02F3/00 - E02F7/00
- E02F9/26—Indicating devices
- E02F9/264—Sensors and their calibration for indicating the position of the work tool
-
- E—FIXED CONSTRUCTIONS
- E02—HYDRAULIC ENGINEERING; FOUNDATIONS; SOIL SHIFTING
- E02F—DREDGING; SOIL-SHIFTING
- E02F9/00—Component parts of dredgers or soil-shifting machines, not restricted to one of the kinds covered by groups E02F3/00 - E02F7/00
- E02F9/26—Indicating devices
- E02F9/264—Sensors and their calibration for indicating the position of the work tool
- E02F9/265—Sensors and their calibration for indicating the position of the work tool with follow-up actions (e.g. control signals sent to actuate the work tool)
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/0101—Head-up displays characterised by optical features
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G5/00—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators
-
- B60K2360/166—
-
- B60K2360/171—
-
- B60K2360/177—
-
- B60K2360/21—
-
- B60K2360/31—
-
- B60K2360/61—
-
- B60K2360/785—
-
- B60K35/81—
-
- G—PHYSICS
- G05—CONTROLLING; REGULATING
- G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
- G05B2219/00—Program-control systems
- G05B2219/30—Nc systems
- G05B2219/45—Nc applications
- G05B2219/45012—Excavator
Definitions
- the present invention relates to an image display system for a work machine, a remote operation system for the work machine, a work machine, and an image display method for the work machine.
- Patent Document 1 a technique for remotely operating a work machine such as a hydraulic excavator is known. Further, Patent Document 2 controls the display position along the combiner surface of the work support information and the depth display position in front of the cab based on the position information of the work implement, and the work support perceived by the operator in the cab. A technique for displaying a projected image of information around a work machine is described.
- An aspect of the present invention aims to suppress a decrease in work efficiency when working using a work machine equipped with a work machine.
- a position calculation unit that obtains the position of a work machine that a work machine has, a display unit that displays a route image corresponding to a route that the work machine will move, and the route The route image corresponding to a portion of the route that is shielded by the work implement from a predetermined position as a viewpoint is shielded by the work implement based on the position of the work implement and the position of the work implement obtained by the position calculation unit.
- an image display system for a work machine including a display processing unit that displays on the display unit in a form different from the route image corresponding to the portion of the route that is not performed.
- the display processing unit converts the route image into an image viewed from the predetermined position and displays the image on the display unit.
- An image display system is provided.
- the image processing apparatus includes an imaging device that is attached to the work machine and images a target, and the display processing unit captures the path image and the imaging device.
- an image display system for a work machine that combines the image of the work machine thus displayed and displays the image on the display unit.
- the image display system for a work machine according to the third aspect, wherein the predetermined position is a position of an imaging device that is attached to the work machine and images a target. .
- the display processing unit is configured to change the route according to a distance between the work implement and the route.
- An image display system for a work machine that changes an image display form is provided.
- a work machine comprising: a work machine image display system according to any one of the first to fifth aspects; and an operating device that operates the work machine.
- a remote control system is provided.
- a work machine including the work machine image display system according to any one of the first to fifth aspects.
- the position of the work machine of the work machine is obtained, and based on the position of the path on which the work machine will move and the position of the work machine, the predetermined position is used as a viewpoint. It is determined whether or not the work machine shields the route, the route image corresponding to the route of the part where the work machine shields the route, and the route corresponding to the part where the work machine does not shield the route.
- an image display method for a work machine that is displayed on a display unit in a form different from an image.
- the aspect of the present invention can suppress a decrease in work efficiency when working using a work machine equipped with a work machine.
- FIG. 1 is a diagram illustrating an image display system for a work machine and a remote operation system for the work machine according to an embodiment.
- FIG. 2 is a diagram illustrating a control system of the hydraulic excavator.
- FIG. 3 is a control block diagram of a processing unit included in the processing device of the image display system.
- FIG. 4 is a diagram illustrating an example of converting a position in the global coordinate system into a position in the imaging apparatus coordinate system.
- FIG. 5 is a diagram for explaining processing for determining whether or not the work machine shields the route when the route image is viewed from a predetermined position.
- FIG. 6 is a diagram illustrating an example of a state in which a route image and a bucket of a work machine are displayed superimposed on a display device.
- FIG. 1 is a diagram illustrating an image display system for a work machine and a remote operation system for the work machine according to an embodiment.
- FIG. 2 is a diagram illustrating a control system of the hydraulic exca
- FIG. 7 is a diagram illustrating an example of a state in which a route image and a bucket of a work machine are displayed superimposed on a display device.
- FIG. 8 is a flowchart illustrating a procedure of the image display method for the work machine according to the embodiment.
- FIG. 9 is a diagram illustrating a control system of a hydraulic excavator according to a modified example of the embodiment.
- FIG. 1 is a diagram illustrating a work machine image display system 100 and a work machine remote control system 101 according to an embodiment.
- An image display system 100 for a work machine (hereinafter, referred to as an image display system 100 as appropriate) is a work object of the excavator 1 when the operator of the work machine remotely operates the excavator 1 which is an example of the work machine. More specifically, the terrain surface that is a target of work by the work machine 2 provided in the excavator 1, that is, the work target WA and the bucket 8 that is the work tool are picked up by the image pickup device 19, and the obtained image is displayed on the display device 52. Display. At this time, the image display system 100 causes the display device 52 to display an image including the bucket 8 and a route image TL corresponding to the route tl when the bucket 8 moves.
- the image display system 100 includes a position calculation unit 51C, a route calculation unit 51R, a display device 52 as a display unit, and a display processing unit 51D.
- a work machine remote operation system 101 (hereinafter referred to as a remote operation system 101 as appropriate) includes an image display system 100 and an operation device 53.
- the position calculation device 23 of the image display system 100 is installed in the excavator 1, and the position calculation unit 51C, the route calculation unit 51R, the display device 52, and the display processing unit 51D are installed in the facility 50.
- the facility 50 is used for remotely operating the excavator 1 and managing the excavator 1.
- the position calculation unit 51C, the route calculation unit 51R, and the display processing unit 51D of the image display system 100 are part of the processing device 51 in the facility 50.
- the processing device 51 includes a processing unit 51P, a storage unit 51M, and an input / output unit 51IO.
- the processing unit 51P is exemplified by a processor such as a CPU (Central Processing Unit) and a GPU (Graphical Processing Unit).
- Examples of the storage unit 51M include a RAM (Random Access Memory), a ROM (Read Only Memory), a hard disk drive, a storage device, or a combination thereof.
- the input / output unit 51IO is an interface circuit for connecting the processing device 51 and an external device.
- a display device 52, an operation device 53, and a communication device 54 are connected to the input / output unit 51IO as external devices.
- the external devices connected to the input / output unit 51IO are not limited to these.
- the processing unit 51P includes a position calculation unit 51C, a route calculation unit 51R, and a display processing unit 51D.
- the position calculation unit 51C obtains the position of the work machine 2 of the excavator 1.
- the position of the work machine 2 includes the position of the work tool that the work machine 2 has, in the embodiment, the bucket 8.
- the route calculation unit 51R obtains a route tl when the work implement 2 moves, that is, a route tl (the route tl that the work implement 2 is scheduled to move) from which the work implement 2 will move.
- a route tl when the bucket 8 that is a part of the work machine 2 moves is obtained, but the route of the bucket 8 is not limited.
- the route tl is a route that becomes a target when the bucket 8 moves. That is, the route tl serves as a target when the operator moves the work implement 2 and thus becomes a target index for supporting the operator.
- the route tl when the work implement 2 moves is, for example, a route that prevents the work implement 2 from contacting an obstacle, a route that can be moved in the shortest, a route that can be moved most efficiently, or a target construction surface on site.
- a route along design information such as a design surface can be used.
- the working machine 2 has a route at the time of hoist turning that lifts the work machine 2 after excavating the object, and a route at the time of hoist turning that turns the work machine 2 while lowering the work machine 2 after loading or waste soil. It may be a route tl along which 2 moves.
- the center in the width direction of the cutting edge 8T of the bucket 8 is set as the reference position of the path tl.
- the present invention is not limited to this, and the left end or the right end of the cutting edge 8T of the bucket 8 may be set as the reference point.
- the reference position of the path tl may be used.
- the display processing unit 51D uses the obtained route tl as a viewpoint to determine the route tl. Finds the first part tl1 which is the part shielded by the work implement 2.
- the display processing unit 51D displays the route image TL of the first portion tl1 where the route tl is shielded by the work implement 2 when viewed from a predetermined position, and the second portion which is a portion where the route tl is not shielded by the work implement 2 It is displayed on the display device 52 in a form different from the route image TL of tl2.
- the work machine 2 includes a bucket 8, an arm 7, and a boom 6 as members constituting the work machine 2. Therefore, the route tl being shielded by the work implement 2 means that the route tl is shielded by the members constituting the work implement 2.
- the route tl is shielded by at least a part of the work machine 2, for example, the bucket 8, the blade 8B of the bucket 8, the arm 7, the boom 6, or the bucket 8 and the arm 7, and the path tl is In any case, the path tl is included in the shielding by the work machine 2.
- the predetermined position is exemplified by the position corresponding to the viewpoint of the operator who operates the hydraulic excavator 1 and the position of the imaging device 19 attached to the hydraulic excavator 1 and imaging the target.
- the position corresponding to the viewpoint of the operator who operates the excavator 1 is exemplified by the position of the eye in a posture where the operator is seated on the driver's seat of the excavator 1 and operates the excavator 1.
- the predetermined position is not limited to the position corresponding to the operator's viewpoint and the position of the imaging device 19, and may be a free viewpoint unrelated to these viewpoints. Examples of the free viewpoint include a bird's-eye viewpoint and an operator's viewpoint arranged in a virtual space.
- the processing unit 51P reads and executes a computer program for realizing the functions of the position calculation unit 51C, the route calculation unit 51R, and the display processing unit 51D from the storage unit 51M, thereby executing the position calculation unit 51C and the route.
- the functions of the calculation unit 51R and the display processing unit 51D are realized.
- the functions of the position calculation unit 51C, the route calculation unit 51R, and the display processing unit 51D may be realized by a processing circuit.
- the processing circuit is dedicated hardware for realizing the functions of the position calculation unit 51C, the route calculation unit 51R, and the display processing unit 51D.
- the processing circuit corresponds to a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a combination thereof.
- the display device 52 displays the path tl generated by the processing device 51 and the image captured by the imaging device 19 of the excavator 1, that is, the image including the bucket 8 in the embodiment.
- the display device 52 is exemplified by a liquid crystal display or a projector, but is not limited thereto.
- the communication device 54 includes an antenna 54A. The communication device 54 communicates with the communication device 25 provided in the excavator 1 to acquire information on the excavator 1 or transmit information to the excavator 1.
- the operating device 53 has a left operating lever 53L installed on the left side of the operator and a right operating lever 53R arranged on the right side of the operator.
- the front-rear and left-right operations correspond to the biaxial operations.
- the operation in the front-rear direction of the right operation lever 53R corresponds to the operation of the boom 6 of the work machine 2 provided in the hydraulic excavator 1.
- the left / right operation of the right operation lever 53R corresponds to the operation of the bucket 8 of the work machine 2.
- the operation in the front-rear direction of the left operation lever 53L corresponds to the operation of the arm 7 of the work machine 2.
- the left / right operation of the left operation lever 53L corresponds to the turning of the swing body 3 of the excavator 1.
- the operation amounts of the left operation lever 53L and the right operation lever 53R are detected by, for example, a potentiometer and a Hall IC, and the processing device 51 is provided with an electromagnetic control valve provided in the hydraulic circuit of the hydraulic excavator 1 based on these detected values.
- a control signal for controlling the signal is generated.
- This signal is sent to the work machine control device 27 via the communication device 54 of the facility 50 and the communication device 25 of the excavator 1.
- the work machine control device 27 controls the work machine 2 by controlling the electromagnetic control valve based on the control signal.
- the electromagnetic control valve will be described later.
- the processing device 51 acquires an input to at least one of the left operation lever 53L and the right operation lever 53R, and generates a command for operating at least one of the work implement 2 and the swing body 3.
- the processing device 51 transmits the generated command to the communication device 25 of the excavator 1 via the communication device 54.
- the work machine control device 27 included in the excavator 1 acquires a command from the processing device 51 via the communication device 25, and operates at least one of the work machine 2 and the swing body 3 according to the command.
- the hydraulic excavator 1 includes a communication device 25, a work machine control device 27, a first posture detection device 32, a second posture detection device 33, an imaging device 19, antennas 21 and 22, and a position calculation device 23.
- the communication device 25 is connected to the antenna 24 and communicates with a communication device 54 provided in the facility 50.
- the work machine control device 27 controls the operation of the work machine 2 and the swing body 3.
- the first attitude detection device 32 detects the attitude of the work machine 2.
- the second attitude detection device 33 detects the attitude of the excavator 1.
- the imaging device 19 is attached to the excavator 1 and images the work implement 2 or the work target WA.
- the antennas 21 and 22 receive radio waves from the positioning satellite 200.
- the position calculation device 23 uses the radio waves received by the antennas 21 and 22 to determine the global positions of the antennas 21 and 22, that is, the positions in the global coordinate system.
- the hydraulic excavator 1 has a vehicle main body 1B and a work implement 2 as main body portions.
- the vehicle main body 1 ⁇ / b> B includes a turning body 3 and a traveling body 5.
- the excavator 1 uses an internal combustion engine such as a diesel engine as an engine that is a power generation device, but the power generation device is not limited to the internal combustion engine.
- the power generation device of the hydraulic excavator 1 may be, for example, a so-called hybrid device in which an internal combustion engine, a generator motor, and a power storage device are combined.
- the power generation device of the hydraulic excavator 1 may not be an internal combustion engine, and may be a device that combines a power storage device and a generator motor.
- the traveling body 5 carries the revolving body 3.
- the traveling body 5 has a crawler belt.
- the work machine 2 is attached to the side of the cab 4 of the revolving structure 3.
- the swivel body 3 is on the front side where the work implement 2 and the operator cab 4 are arranged, and is on the side where the counterweight is arranged.
- the front-rear direction of the revolving structure 3 is the direction of the Xm axis.
- the left side toward the front is the left of the revolving unit 3, and the right side toward the front is the right of the revolving unit 3.
- the left-right direction of the revolving structure 3 is also referred to as the width direction or the direction of the Ym axis.
- the excavator 1 or the vehicle main body 1 ⁇ / b> B has the traveling body 5 side on the basis of the revolving body 3, and the revolving body 3 side on the basis of the traveling body 5.
- the vertical direction of the revolving structure 3 is the direction of the Zm axis.
- the lower side is the vertical direction, that is, the gravity direction side
- the upper side is the opposite side of the vertical direction.
- the work machine 2 includes a boom 6, an arm 7, a bucket 8 as a work tool, a boom cylinder 10, an arm cylinder 11, and a bucket cylinder 12.
- the bucket 8 has a plurality of blades 8B.
- the blade tip 8T is the tip of the blade 8B.
- the bucket 8 is not limited to one having a plurality of blades 8B.
- the bucket 8 may be a tilt bucket.
- the working machine 2 may include, as a work tool, a rock drilling attachment provided with a slope bucket or a chip for rock drilling instead of the bucket 8.
- the boom cylinder 10, the arm cylinder 11, and the bucket cylinder 12 are hydraulic cylinders that are driven by the pressure of hydraulic oil discharged from the hydraulic pump.
- the boom cylinder 10 drives the boom 6.
- the arm cylinder 11 drives the arm 7.
- the bucket cylinder 12 drives the bucket 8.
- the antennas 21 and 22 and the antenna 24 are attached to the upper part of the revolving unit 3.
- the antennas 21 and 22 are used to detect the position of the excavator 1.
- the antennas 21 and 22 are electrically connected to the position calculation device 23 shown in FIG.
- Antennas 21 and 22 are antennas for GNSS (Real Time Kinematic-Global Navigation Satellite Systems, GNSS means Global Navigation Satellite System).
- GNSS Real Time Kinematic-Global Navigation Satellite Systems
- the antennas 21 and 22 are arranged apart from each other by a certain distance along a direction parallel to the Ym axis.
- the antennas 21 and 22 receive GNSS radio waves from the positioning satellite 200 and output signals according to the received GNSS radio waves.
- the antennas 21 and 22 may be antennas for GPS (Global Positioning System).
- the position calculation device 23 is a position detection device that detects the position of the excavator 1.
- the position calculation device 23 detects the position of the excavator 1 using GNSS.
- the imaging device 19 images the work target WA shown in FIG. 1, it is preferable to acquire information from the widest work target WA as much as possible. For this reason, in the embodiment, the imaging device 19 is installed above the cab 4 of the revolving structure 3.
- the place where the imaging device 19 is installed is not limited to above the driver's seat.
- the imaging device 19 may be installed inside and above the cab 4.
- the imaging surface 19L faces the front of the revolving unit 3.
- the imaging device 19 is a monocular camera including an image sensor such as a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor, but is not limited thereto.
- the coordinate system of the excavator 1 is a vehicle body coordinate system (Xm, Ym, Zm), and the coordinate system of the imaging device 19 is an imaging device coordinate system (Xs, Ys, Zs).
- the GNSS or GPS coordinate system is a global coordinate system (Xg, Yg, Zg).
- the global coordinate system (Xg, Yg, Zg) is a coordinate system based on the origin fixed on the earth.
- the vehicle body coordinate system (Xm, Ym, Zm) is a coordinate system based on the origin fixed to the vehicle body 1B, in the embodiment, the swing body 3.
- the origin of the vehicle body coordinate system (Xm, Ym, Zm) is, for example, the center of the swing circle of the swing body 3.
- the center of the swing circle exists on the turning center axis of the turning body 3.
- the Zm axis of the vehicle body coordinate system (Xm, Ym, Zm) is an axis that becomes the turning center axis of the turning body 3.
- the Xm axis is an axis that extends in the front-rear direction of the revolving structure 3 and is orthogonal to the Zm axis.
- the Xm axis is a reference axis in the front-rear direction of the swing body 3.
- the Ym axis is an axis extending in the width direction of the revolving structure 3 and orthogonal to the Zm axis and the Xm axis.
- the origin of the imaging device coordinate system is, for example, the center of the imaging device 19.
- the center of the imaging device 19 can be the optical center of the imaging device 19 or the center of the light receiving surface of the image sensor included in the imaging device 19.
- the Xs axis of the imaging device coordinate system is an axis that passes through the optical center of the imaging device 19 and extends in a direction orthogonal to the imaging surface 19L.
- the Ys axis is an axis orthogonal to the Xs axis.
- the Zs axis is an axis orthogonal to both the Xs axis and the Ys axis.
- FIG. 2 is a diagram illustrating a control system 1 ⁇ / b> S of the hydraulic excavator 1.
- the control system 1S includes a communication device 25, a sensor controller 26, a work machine control device 27, an imaging device 19, a position calculation device 23, a first posture detection device 32, a second posture detection device 33, and a hydraulic pressure. And a system 36.
- the communication device 25, the sensor controller 26, and the work machine control device 27 are connected by a signal line 35. With such a structure, the communication device 25, the sensor controller 26, and the work machine control device 27 can exchange information with each other via the signal line 35.
- the signal line for transmitting information in the control system 1S is exemplified by an in-vehicle signal line such as CAN (Controller Area Network).
- the sensor controller 26 includes a processor such as a CPU (Central Processing Unit) and storage devices such as a RAM and a ROM.
- the sensor controller 26 receives the detection value of the position calculation device 23, the information of the image captured by the imaging device 19, the detection value of the first posture detection device 32, and the detection value of the second posture detection device 33.
- the sensor controller 26 transmits the input detection value and image information to the processing device 51 of the facility 50 shown in FIG. 1 via the signal line 35 and the communication device 25.
- the work machine control device 27 includes a processor such as a CPU (Central Processing Unit) and a storage device such as a RAM (Random Access Memory) and a ROM (Read Only Memory).
- the work machine control device 27 acquires a command generated by the processing device 51 of the facility 50 for operating at least one of the work machine 2 and the swivel body 3 via the communication device 25.
- the work machine control device 27 controls the electromagnetic control valve 28 of the hydraulic system 36 based on the acquired command.
- the hydraulic system 36 includes an electromagnetic control valve 28, a hydraulic pump 29, and hydraulic actuators such as a boom cylinder 10, an arm cylinder 11, a bucket cylinder 12, and a swing motor 30.
- the hydraulic pump 29 is driven by the engine 31 and discharges hydraulic oil for operating the hydraulic actuator.
- the work machine control device 27 controls the flow rate of hydraulic oil supplied to the boom cylinder 10, the arm cylinder 11, the bucket cylinder 12, and the swing motor 30 by controlling the electromagnetic control valve 28. In this way, the work machine control device 27 controls the operations of the boom cylinder 10, the arm cylinder 11, the bucket cylinder 12, and the turning motor 30.
- the sensor controller 26 acquires the detection values of the first stroke sensor 16, the second stroke sensor 17, and the third stroke sensor 18.
- the first stroke sensor 16 is provided in the boom cylinder 10
- the second stroke sensor 17 is provided in the arm cylinder 11
- the third stroke sensor 18 is provided in the bucket cylinder 12.
- the first stroke sensor 16 detects the boom cylinder length which is the length of the boom cylinder 10 and outputs it to the sensor controller 26.
- the second stroke sensor 17 detects the arm cylinder length, which is the length of the arm cylinder 11, and outputs it to the sensor controller 26.
- the third stroke sensor 18 detects the bucket cylinder length, which is the length of the bucket cylinder 12, and outputs it to the sensor controller 26.
- the attitude of the work implement 2 is determined.
- the first stroke sensor 16, the second stroke sensor 17, and the third stroke sensor 18 that detect these correspond to the first posture detection device 32 that detects the posture of the work implement 2.
- position detection apparatus 32 is not limited to the 1st stroke sensor 16, the 2nd stroke sensor 17, and the 3rd stroke sensor 18, An angle detector may be sufficient.
- the sensor controller 26 is a boom angle that is an inclination angle of the boom 6 with respect to a direction (Zm axis direction) orthogonal to the horizontal plane in the vehicle body coordinate system that is the coordinate system of the excavator 1 from the boom cylinder length detected by the first stroke sensor 16. Is calculated.
- the sensor controller 26 calculates an arm angle that is an inclination angle of the arm 7 with respect to the boom 6 from the arm cylinder length detected by the second stroke sensor 17.
- the sensor controller 26 calculates a bucket angle that is an inclination angle of the bucket 8 with respect to the arm 7 from the bucket cylinder length detected by the third stroke sensor 18.
- the inclination angles of the boom 6, the arm 7, and the bucket 8 are information indicating the posture of the work machine 2. That is, the sensor controller 26 obtains information indicating the posture of the work machine 2.
- the sensor controller 26 transmits the calculated inclination angle to the processing device 51 of the facility 50 shown in FIG. 1 via the signal line 35 and the communication device 25.
- the position calculation device 23 has a processor such as a CPU and a storage device such as a RAM and a ROM.
- the position calculation device 23 obtains the position of the excavator 1. Specifically, the position calculation device 23 detects the positions of the antennas 21 and 22 and the orientation of the revolving unit 3 in the global coordinate system (Xg, Yg, Zg) using the signals acquired from the antennas 21 and 22, Output.
- the orientation of the revolving structure 3 represents the direction of the revolving structure 3 in the global coordinate system.
- the orientation of the revolving structure 3 can be expressed by, for example, the front-rear direction of the revolving structure 3 around the Zg axis of the global coordinate system.
- the azimuth angle is a rotation angle of the reference axis in the front-rear direction of the revolving unit 3 around the Zg axis in the global coordinate system.
- the azimuth of the revolving unit 3 is represented by the azimuth angle.
- the second attitude detection device 33 is an IMU (Inertial Measurement Unit) in the embodiment.
- the second attitude detection device 33 detects the operation and attitude of the excavator 1.
- position detection apparatus 33 may be called IMU33.
- the operation of the hydraulic excavator 1 includes at least one of the operation of the swing body 3 and the operation of the traveling body 5.
- the posture of the excavator 1 is represented by the roll angle, pitch angle, and yaw angle of the excavator 1.
- the second attitude detection device 33 detects and outputs the angle or angular velocity and acceleration of the excavator 1.
- the angles of the excavator 1 detected by the second attitude detection device 33 are the roll angle, pitch angle, and yaw angle of the excavator 1.
- processing device 51 When the operator of the hydraulic excavator 1 remotely operates the hydraulic excavator 1 using the operation device 53 while visually recognizing the display device 52 of the remote operation system 101, the processing device 51 serves as a guide serving as an index of remote operation by the operator.
- the route image TL is displayed on the display device 52.
- the screen of the display device 52 is a two-dimensional screen, and it is difficult to convey the depth to the operator.
- the processing device 51 looks from a predetermined position, for example, a position corresponding to the viewpoint of the operator or the position of the imaging device 19 of the hydraulic excavator 1. Information on whether or not the route tl is blocked by the work implement 2 is displayed on the display device 52. Specifically, the processing device 51 obtains a first portion where the path tl is shielded by the work implement 2 when viewed from a predetermined position. Then, the processing device 51 displays the route image TL of the first part in which the route tl is shielded by the work implement 2 in a form different from the route image TL of the second portion in which the route tl is not shielded by the work implement 2.
- the processing device 51 when viewed from a predetermined position, does not display the first portion of the route image TL in which the route tl is shielded by the work implement 2 and the route tl is not shielded by the work implement 2. A two-part route image TL is displayed.
- the processing device 51 can inform the operator of information on whether the work implement 2 is on the imaging device 19 side or the back side of the excavator 1 with respect to the path t1, that is, depth information. Therefore, it is possible to suppress a decrease in work efficiency in remote operation.
- FIG. 3 is a control block diagram of the processing unit 51P included in the processing device 51 of the image display system 100.
- FIG. 4 is a diagram illustrating an example of converting a position in the global coordinate system into a position in the imaging apparatus coordinate system.
- FIG. 5 is a diagram for explaining processing for determining whether or not the route tl is blocked by the work implement 2 when the route image TL is viewed from a predetermined position.
- 6 and 7 are diagrams illustrating an example of a state in which the route image TL and the bucket 8 of the work machine 2 are displayed superimposed on the display device 52.
- the processing unit 51P receives the vehicle information IFm and supplies the received vehicle information IFm to the position calculation unit 51C, the route calculation unit 51R, and the display processing unit 51D.
- the display processing unit 51D includes a viewpoint conversion unit 51DT, a work machine area calculation unit 51DM, a route drawing unit 51DR, a shielding calculation unit 51DC, and an image composition unit 51DS.
- the vehicle information IFm includes the position and posture of the excavator 1 and the posture of the work machine 2 included in the excavator 1.
- the position Pg of the excavator 1 is at least one position of the antenna 21 and the antenna 22 in the global coordinate system detected by the position calculation device 23.
- the attitude of the excavator 1 is the roll angle ⁇ r, pitch angle ⁇ p, and yaw angle ⁇ y of the excavator 1 detected by the second attitude detection device 33.
- the yaw angle may be an azimuth angle obtained from the relative position of the two antennas 21 and 22 by the position calculation device 23.
- the posture of the work implement 2 is detected by the first stroke sensor 16, the second stroke sensor 17, and the third stroke sensor 18 which are the first posture detection device 32.
- the posture of the work machine 2 is the boom angle ⁇ b, the arm angle ⁇ a, and the bucket angle ⁇ k.
- the vehicle information IFm is sent from the sensor controller 26 of the excavator 1 to the communication device 54 of the facility 50 via the communication device 25.
- the communication device 54 that has received the vehicle information IFm gives the vehicle information IFm to the input / output unit 51IO of the processing device 51.
- the input / output unit 51IO provides the received vehicle information IFm to the processing unit 51P.
- the position calculation unit 51C, the route calculation unit 51R of the processing unit 51P, and the viewpoint conversion unit 51DT of the display processing unit 51D each perform processing using the received vehicle information IFm.
- the position calculation unit 51C obtains the position of the work implement 2 using information related to the work implement 2 in the received vehicle information IFm.
- the information regarding the work machine 2 is the boom angle ⁇ b, the arm angle ⁇ a, and the bucket angle ⁇ k of the work machine 2.
- the dimensions of the work implement 2, for example, the length of the boom 6, the length of the arm 7, and the length from the rotation center of the bucket 8 to the cutting edge 8T are stored in advance in the storage unit 51M of the processing device 51 as a model of the work implement 2. ing.
- the position of the cutting edge 8T of the bucket 8 when obtaining the position of the cutting edge 8T of the bucket 8 as the position of the work machine 2, for example, the boom angle ⁇ b, the arm angle ⁇ a and the bucket angle ⁇ k of the work machine 2, the length of the boom 6, the length of the arm 7, and the bucket
- the positional relationship between the base end of the boom 6 on the revolving structure 3 side and the cutting edge 8T can be determined by the length from the rotation center 8 to the cutting edge 8T.
- the position computing unit 51C The position of the cutting edge 8T of the bucket 8 in the global coordinate system can be obtained from the position 1 Pg, information on the work machine 2, the dimensions of the work machine 2, and the orientation of the swivel body 3.
- the orientation of the swing body 3 may be obtained from these positions obtained by the antennas 21 and 22, or the yaw angle of the swing body 3 detected by the IMU 33 may be used.
- the position of the cutting edge 8T of the bucket 8 that is the position of the work machine 2 is obtained from the position of the work machine 2 and the attitude of the excavator 1, more specifically, the position of the work machine 2 and the attitude of the swivel body 3. .
- the position of the work machine 2 is obtained from the dimensions of the work machine 2 and the posture of the work machine 2.
- the position calculation unit 51 ⁇ / b> C obtains the positions of the bucket 8, the arm 7, and the boom 6 that the work machine 2 has as the position of the work machine 2.
- the position calculation unit 51 ⁇ / b> C obtains the position of the work implement 2 using, for example, the model of the work implement 2 based on the size and shape information of the work implement 2.
- the dimensions and shape of the work machine 2 are included in the design information of the work machine 2 or a CAD (Computer Aided Design) model.
- the position calculation unit 51C has obtained the cutting edge 8T of the bucket 8 as the position of the work machine 2.
- the position calculation unit 51C obtains a plurality of representative positions from which the outer shape of the work machine 2 can be grasped, and sets the position within the range surrounded by the plurality of representative positions to the work machine 2. It is good also as a position.
- the position of the work machine 2 includes at least one of the position of the bucket 8, the position of the arm 7, and the position of the boom 6.
- the position of the cutting edge 8T described above is a part of the position of the bucket 8.
- the position calculation unit 51 ⁇ / b> C obtains the position of the bucket 8, the position of the arm 7, and the position of the boom 6 as the position of the work machine 2.
- the position calculation unit 51C changes the target for obtaining the position of the work machine 2 among the work machines 2 according to the part of the work machine 2 occupying the screen of the display device 52, the position of the path tl, the processing load, and the like. Also good. For example, when the bucket 8 and a part of the arm 7 are shown on the screen of the display device 52, the position calculation unit 51C may obtain the position of the bucket 8 and the position of the arm 7 as the position of the work implement 2. . By doing in this way, the load of the process which 51C of position calculating parts perform is reduced.
- the position calculation unit 51C may obtain only the position of the bucket 8 of the work implement 2 as the position of the work implement 2 when the processing load becomes excessive when all the positions of the bucket 8, the arm 7 and the boom 6 are obtained. . By doing so, the load of processing executed by the position calculation unit 51C is reduced, so that the load of hardware resources of the image display system 100 is reduced.
- the route calculation unit 51R obtains a route when the work implement 2 moves, in the embodiment, a route when the bucket 8 moves, based on the received vehicle information IFm. As described above, this route is a route targeted when the work machine 2 moves. In obtaining the route, information on the current position and posture of the excavator 1 is necessary. The route calculation unit 51R obtains a route based on the current position Pg and posture of the excavator 1 included in the vehicle information IFm. When the route calculation unit 51R obtains a route, the route calculation unit 51R includes the hydraulic excavator 1 obtained by the imaging device 19 of the excavator 1, or a distance detection device such as a stereo camera or a laser scanner mounted on the excavator 1. You may use information around. For example, when the excavator 1 loads a load on the dump truck, the route calculation unit 51R can obtain a route in which the work machine 2 does not interfere with the dump truck. This route is a route in the global coordinate system.
- the viewpoint conversion unit 51DT obtains a conversion matrix for converting the path image TL and the position of the work machine 2 into a predetermined position, for example, a position corresponding to the line of sight of the operator or an image viewed from the position of the imaging device 19.
- a predetermined position for example, a position corresponding to the line of sight of the operator or an image viewed from the position of the imaging device 19.
- the position Pgk (xg, yg, zg) of the work machine 2 in the global coordinate system (Xg, Yg, Zg) is the position of the work machine 2 in the imaging device coordinate system (Xs, Ys, Zs).
- An example of conversion to Psk (xs, ys, zs) will be described.
- the viewpoint conversion unit 51DT changes the position Pgk (xg, yg, zg) of the work machine 2 in the global coordinate system to the position Pmk (xm, ym, zm) of the work machine 2 in the vehicle body coordinate system (Xm, Ym, Zm).
- a first conversion matrix Rg for conversion is obtained.
- the position Pgk of the work machine 2 in the global coordinate system is converted to the position Pmk of the work machine 2 in the vehicle body coordinate system.
- the first transformation matrix Rg is a 3 ⁇ 4 matrix including the roll angle ⁇ r, the pitch angle ⁇ p, the yaw angle ⁇ y of the excavator 1 and the translational components of the global coordinate system and the vehicle body coordinate system.
- the roll angle ⁇ r, pitch angle ⁇ p, and yaw angle ⁇ y of the hydraulic excavator 1 are obtained from the vehicle information IFm.
- the translational component between the global coordinate system and the vehicle body coordinate system can be obtained from the position Pg of the hydraulic excavator 1 included in the vehicle information IFm and the positional relationship between the position Pg of the hydraulic excavator 1 and the vehicle body coordinate system.
- the viewpoint conversion unit 51DT obtains a second conversion matrix Rm for converting the position Pmk of the work machine 2 in the vehicle body coordinate system to the position Psk of the work machine 2 in the imaging device coordinate system. For this reason, by calculating the product of the position Pmk and the second conversion matrix Rm, the position Pmk of the work machine 2 in the vehicle body coordinate system is converted to the position Psk of the work machine 2 in the imaging device coordinate system.
- the second transformation matrix Rm includes an angle ⁇ representing the displacement of the imaging device 19 around the Xs axis, an angle ⁇ representing the displacement of the imaging device 19 around the Ys axis, an angle ⁇ representing the displacement of the imaging device 19 around the Zs axis, And a 3 ⁇ 4 matrix including translational components of the vehicle body coordinate system and the imaging apparatus coordinate system.
- the angles ⁇ , ⁇ , and ⁇ are obtained in advance.
- the translational component between the vehicle body coordinate system and the imaging device coordinate system is obtained in advance from the positional relationship between the two in the hydraulic excavator 1.
- the work machine area calculation unit 51DM uses the first conversion matrix Rg and the second conversion matrix Rm obtained by the viewpoint conversion unit 51DT to calculate the position Pgk of the work machine 2 in the global coordinate system in the imaging device coordinate system. Convert to position Psk.
- the path drawing unit 51DR uses the first conversion matrix Rg and the second conversion matrix Rm obtained by the viewpoint conversion unit 51DT to convert the path tl or the path image TL in the global coordinate system into the path tl in the imaging apparatus coordinate system. Alternatively, it is converted into a route image TL.
- the shielding calculation unit 51DC is based on a route image TL corresponding to the route tl or the route tl of the work implement 2 and the position Psk of the work implement 2, for example, a predetermined position of the imaging device 19.
- the route tl is viewed from the position, the first portion where the route tl is shielded by the work implement 2 is obtained.
- the route tl of the work machine 2 or the route image TL corresponding to the route tl is obtained by the route calculation unit 51R and converted into the imaging device coordinate system by the first conversion matrix Rg and the second conversion matrix Rm.
- a route tl or a route image TL is obtained by the route calculation unit 51R and converted into the imaging device coordinate system by the first conversion matrix Rg and the second conversion matrix Rm.
- the position Psk of the work machine 2 is a position obtained by the position calculation unit 51C and converted into the imaging apparatus coordinate system by the first conversion matrix Rg and the second conversion matrix Rm.
- the shielding calculation unit 51DC displays the route image TL of the first part where the route tl is shielded by the work implement 2 on the display device 52 in a form different from the route image TL of the second portion where the route tl is not shielded by the work implement 2. Generate information to be displayed.
- the first portion tl1 where the route image TL corresponding to the route tl or the route tl of the bucket 8 or the route tl in this example is blocked by the bucket 8 is a range from the position a to the position b.
- this range is referred to as a range ab as appropriate.
- the first portion tl1 in which the path tl or the path image TL corresponding to the path tl is shielded by the bucket 8 is obtained when the bucket 8 and the path tl or the path tl are viewed from the position of the imaging device 19 that is a predetermined position. 8 is a portion where the path tl or the path image TL is blocked.
- the shielding calculation unit 51DC uses, for example, a process called a Z buffer or a W buffer (see Catmull, E. A Subdivision Algorithm for Computer Display of Curved Surfaces. 1974) in computer graphics technology to work the route tl.
- the first part tl1 shielded by the machine 2 and the second part tl2 not shielded can be obtained.
- the shielding calculation unit 51DC of the display processing unit 51D causes the display device 52 to display the route image TL existing in the range ab in a form different from the route image TL outside the range ab.
- the shielding calculation unit 51DC displays the route image TL in the range ab transparent or semi-transparent and adds a color to the route image TL outside the range ab, or displays the color of the route image TL in the range ab.
- the color of the route image TL outside the range ab is displayed differently, or the line type of the route image TL within the range ab is different from the line type of the route image TL outside the range ab.
- the line type includes the thickness of the line.
- Displaying the route image TL in the range ab in a transparent manner means that an image of the work machine 2 having a different form from the route image TL outside the range ab is displayed in the portion of the route image TL in the range ab.
- the route image TL in the range ab is displayed transparently, the image of the bucket 8 is displayed in the portion of the route image TL in the range ab.
- the imaging device 19 When the imaging device 19 is at the first position PLa and the route tl and the route image TL are represented by a one-dot chain line, that is, when the route tl exists between the work machine 2 and the imaging device 19, the bucket There is no portion where the path 8 tl or the path image TL corresponding to the path tl is blocked by the bucket 8. That is, when the bucket 8 and the path tl or path tl are viewed from the position of the imaging device 19 that is a predetermined position, there is no portion where the path tl or the path image TL is blocked by the bucket 8. In this case, the shielding calculation unit 51DC displays the route image TL on the display device 52 in the same form.
- the first portion tl1 is a range from the position c to the position d.
- this range is referred to as a range cd as appropriate.
- the second portion tl2 where the bucket 8 does not shield the path tl of the bucket 8 or the path image TL corresponding to the path tl.
- an operator who actually gets on the excavator 1 may wear a head-mounted display, and a route image TL may be displayed on the head-mounted display.
- the position of the operator's viewpoint is changed by an operation such as moving the head
- the position of the head mounted display is also changed.
- the shielding calculation unit 51DC may obtain a portion where the path tl is shielded by the work implement 2 based on the changed position.
- the image composition unit 51DS synthesizes the route image TL obtained by the shielding calculation unit 51DC and the image of the work machine 2 imaged by the imaging device 19 and causes the display device 52 to display the synthesized image.
- the route image TL is an image in a three-dimensional space, but the route image TL displayed on the display device 52 is a two-dimensional image.
- the image of the work machine 2 captured by the imaging device 19 is also a two-dimensional image. Accordingly, the image composition unit 51DS performs perspective projection conversion in which a path image TL defined in a three-dimensional space, that is, the imaging apparatus coordinate system, is projected onto a two-dimensional plane, and the converted path image TL is captured by the imaging apparatus. 19 is combined with the image of the work machine 2 imaged by 19 and displayed on the display device 52.
- FIGS. 6 shows a state in which the bucket 8 is present behind the path tl when viewed from the position corresponding to the operator's viewpoint or the position of the imaging device 19.
- the route image TL does not have a portion displayed in a different form.
- FIG. 7 shows an example in which the bucket 8 is present in front of the route tl and the portion where the route tl is shielded by the bucket 8 is displayed transparently.
- an image of the bucket 8 having a different form from the portion where the route tl is not shielded by the bucket 8 is displayed in the portion where the route tl is shielded by the bucket 8.
- the image display system 100 specifically the processing device 51, can inform the operator of whether the work machine 2 is on the imaging device 19 side or the back side of the route tl, so that the remote Reduction in work efficiency in operation can be suppressed.
- FIG. 8 is a flowchart illustrating a procedure of the image display method for the work machine according to the embodiment.
- the image display method of the work machine is realized by the image display system 100.
- the position calculation unit 51 ⁇ / b> C of the processing device 51 obtains the position of the work machine 2.
- the display processing unit 51 ⁇ / b> D of the processing device 51 determines whether the route tl is blocked by the work implement 2 when viewed from a predetermined position.
- step S103 the display processing unit 51D displays the route image TL of the shielded portion in a form different from the route image TL of the non-shielded portion. It is displayed on the display device 52.
- Step S104 the display processing unit 51D causes the display device 52 to display all the route images TL in the same form.
- FIG. 9 is a diagram illustrating a control system 1Sa of the excavator 1 according to a modification of the embodiment.
- the image display system 100 and the remote operation system 101 described above remotely operate the hydraulic excavator 1 using the operation device 53 of the facility 50 shown in FIG.
- a display device 52a is provided in the cab 4 of the excavator 1, and a route image TL is displayed on the display device 52a in order to assist the operation of the operator of the excavator 1.
- a processing device 51 is connected to the signal line 35 of the control system 1S described above.
- the operation device 53a is connected to the signal line 35 of the control system 1S described above via the work implement control device 17.
- a display device 52 a is connected to the processing device 51.
- the processing device 51 provided in the control system 1Sa has the same function as the processing device 51 provided in the facility 50 shown in FIG. 1 in the image display system 100 and the remote operation system 101 described above.
- the processing device 51 and the display device 52a constitute an image display system 100a according to a modification.
- the display device 52a of the control system 1Sa is a head-up display that projects an image on the front window FG of the driver's seat.
- the display device 52a displays the route image TL at a position that matches the bucket 8 that can be seen through the front window FG.
- the display device 52a is not limited to a head-up display, and may be a normal display.
- the operating device 53a is a device for operating the excavator 1, and includes a left operating lever 53La and a right operating lever 53Ra.
- the operating device 53a may be a pilot hydraulic system or an electric system.
- the image display system 100a uses the route image TL displayed on the front window FG to inform the operator who operates the excavator 1 whether the work implement 2 is closer to the operator side than the route tl. It is possible to inform the operator of the information on the side.
- the image display system 100a can suppress a reduction in work efficiency. The further away the bucket 8 is from the vehicle body 1B of the excavator 1, the more difficult it is for the operator to grasp the perspective. In such a case, the image display system 100a is effective in suppressing a reduction in work efficiency.
- the image display systems 100 and 100a obtain the first part where the work machine 2 shields the path tl when the path tl is viewed from a predetermined position, and the path image TL corresponding to the first part. Is displayed on the display devices 52 and 52a in a form different from the route image TL corresponding to the second portion where the work machine 2 does not block the route tl.
- the image display systems 100 and 100a may display the distance between the route image TL and the work implement 2 on the display devices 52 and 52a in addition to the route image TL.
- the processing device 51 may change the display form of the route image TL in accordance with the distance between the work machine 2 and the route tl. For example, the processing device 51 changes the color of the route image TL as the distance between the work implement 2 and the route tl increases, or the brightness of the route image TL as the distance between the work implement 2 and the route tl increases. Or the route image TL can be made thicker as the distance between the work implement 2 and the route tl increases.
- the position calculation unit 51C obtains a plurality of representative positions where the outer shape of the work machine 2 can be grasped as the position of the work machine 2. In addition to this, the position calculation unit 51C obtains the position of the plane corresponding to the opening of the bucket 8 as the position of the work machine 2, determines whether the work machine 2 blocks the path tl, and the path image TL. You may use for the process which changes a display form.
- the position of the work machine 2 is not limited to that exemplified in the embodiment and the modification.
- the work tool may be other than the bucket 8.
- a gripping device that grips an object may be a work tool.
- the position calculation unit 51C may determine whether the work tool and an object gripped by the work tool shield the path tl.
- a distance detection device such as a stereo camera or a laser scanner is attached to the excavator 1, and the position of the object held by the work tool is detected by the distance detection device.
- the position calculation unit 51C obtains the position of the object held by the work tool from the position information detected by the distance detection device.
- the position calculation unit 51C determines, for example, the position of a portion whose distance from the imaging device 19 is within a predetermined range centering on the position of the gripping portion of the work tool. Suppose that the position of the object held by the work tool. By such processing, the position calculation unit 51C obtains the position of the work tool and the object gripped by the work tool, and the work tool and the object gripped by the work tool shield the path tl based on the obtained position. It is determined whether or not to do.
- the position calculation unit 51 ⁇ / b> C sets the position of the portion raised from the opening of the bucket 8 to the bucket 8 by a process similar to the process for obtaining the position of the object held by the work tool. It can be obtained as the position of the loaded object.
- the position calculation unit 51C obtains the position of the object loaded on the bucket 8 and determines whether the bucket 8 and the object loaded on the bucket 8 block the path tl based on the obtained position. Good. By such processing, the image display systems 100 and 100a can display more realistic images on the display devices 52 and 52a.
- the processing device 51 detects the posture of the work machine 2 and the first operation of the operation devices 53 and 53a by the operator, displays the route image TL on the display devices 52 and 52a, and the second of the operation devices 53 and 53a.
- the display of the route image TL may be turned off.
- the route image TL is erased, so that the visibility of the operator is improved.
- Examples of the first operation include an operation of turning the swing body 3 in a state where the bucket 8 is held at a low position.
- An example of the second operation is a dump operation (a soil discharge operation) for discharging the load in the bucket 8.
- the work machine is not limited to the hydraulic excavator 1 and may be another work machine having a work machine such as a wheel loader or a bulldozer.
Abstract
Description
図1は、実施形態に係る作業機械の画像表示システム100及び作業機械の遠隔操作システム101を示す図である。作業機械の画像表示システム100(以下においては適宜、画像表示システム100と称する)は、作業機械のオペレータが、作業機械の一例である油圧ショベル1を遠隔操作する際に、油圧ショベル1の作業対象、より詳細には油圧ショベル1が備える作業機2による作業の対象である地形面、すなわち作業対象WA及び作業具であるバケット8を撮像装置19で撮像し、得られた画像を表示装置52に表示させる。このとき、画像表示システム100は、バケット8を含む画像及びバケット8が移動する際の経路tlに対応する経路画像TLを表示装置52に表示させる。
油圧ショベル1は、通信装置25と、作業機制御装置27と、第1姿勢検出装置32と、第2姿勢検出装置33と、撮像装置19と、アンテナ21,22と、位置演算装置23とを備える。通信装置25は、アンテナ24に接続されており、施設50に備えられた通信装置54と通信する。作業機制御装置27は、作業機2及び旋回体3の動作を制御する。第1姿勢検出装置32は、作業機2の姿勢を検出する。第2姿勢検出装置33は、油圧ショベル1の姿勢を検出する。撮像装置19は、油圧ショベル1に取り付けられて、作業機2を撮像したり、作業対象WAを撮像したりする。アンテナ21,22は、測位衛星200からの電波を受信する。位置演算装置23は、アンテナ21,22が受信した電波を用いて、アンテナ21,22のグローバル位置、すなわちグローバル座標系における位置を求める。
実施形態において、油圧ショベル1の座標系は車体座標系(Xm,Ym,Zm)であり、撮像装置19の座標系は撮像装置座標系(Xs,Ys,Zs)である。GNSS又はGPSの座標系は、グローバル座標系(Xg,Yg,Zg)である。グローバル座標系(Xg,Yg,Zg)は、地球に固定された原点を基準とした座標系である。
図2は、油圧ショベル1の制御系1Sを示す図である。制御系1Sは、通信装置25と、センサコントローラ26と、作業機制御装置27と、撮像装置19と、位置演算装置23と、第1姿勢検出装置32と、第2姿勢検出装置33と、油圧システム36と、を備える。通信装置25と、センサコントローラ26と、作業機制御装置27とは、信号線35によって接続されている。このような構造により、通信装置25と、センサコントローラ26と、作業機制御装置27とは、信号線35を介して相互に情報をやり取りすることができる。制御系1S内で情報を伝達する信号線は、CAN(Controller Area Network)のような車内信号線が例示される。
油圧ショベル1のオペレータが、遠隔操作システム101の表示装置52を視認しながら操作装置53を用いて油圧ショベル1を遠隔操作する場合、処理装置51は、オペレータによる遠隔操作の指標となるガイドとして、経路画像TLを表示装置52に表示させる。表示装置52の画面は2次元画面であり、奥行きをオペレータに伝えることは難しい。
図8は、実施形態に係る作業機械の画像表示方法の手順を示すフローチャートである。作業機械の画像表示方法は、画像表示システム100によって実現される。ステップS101において、処理装置51の位置演算部51Cは、作業機2の位置を求める。ステップS102において、処理装置51の表示処理部51Dは、所定の位置から見た場合に、経路tlが作業機2によって遮蔽されるかを判定する。経路tlが作業機2によって遮蔽される場合(ステップS102、Yes)、ステップS103において、表示処理部51Dは、遮蔽される部分の経路画像TLを、遮蔽されない部分の経路画像TLとは異なる形態で表示装置52に表示させる。経路tlが作業機2によって遮蔽されない場合(ステップS102、No)、ステップS104において、表示処理部51Dは、すべての経路画像TLを、同一の形態で表示装置52に表示させる。
図9は、実施形態の変形例に係る油圧ショベル1の制御系1Saを示す図である。前述した画像表示システム100及び遠隔操作システム101は、図1に示される施設50の操作装置53を用いて油圧ショベル1を遠隔操作する。変形例は、油圧ショベル1の運転室4内に表示装置52aが備えられており、油圧ショベル1のオペレータの操作を補助するために、経路画像TLが表示装置52aに表示される。
1B 車両本体
1S,1Sa 制御系
2 作業機
3 旋回体
5 走行体
16 第1ストロークセンサ
17 第2ストロークセンサ
18 第3ストロークセンサ
19 撮像装置
19L 撮像面
21,22 アンテナ
23 位置演算装置
25 通信装置
26 センサコントローラ
27 作業機制御装置
32 第1姿勢検出装置
33 第2姿勢検出装置
36 油圧システム
50 施設
51 処理装置
51C 位置演算部
51D 表示処理部
51DC 遮蔽演算部
51DM 作業機領域演算部
51DR 経路描画部
51DS 画像合成部
51DT 視点変換部
51IO 入出力部
51M 記憶部
51P 処理部
51R 経路演算部
52,52a 表示装置
53,53a 操作装置
100,100a 画像表示システム
101 遠隔操作システム
FG フロントウィンドウ
IFm 車両情報
Rg 第1の変換行列
Rm 第2の変換行列
TL 経路画像
Claims (8)
- 作業機械が有する作業機の位置を求める位置演算部と、
前記作業機が移動するであろう経路に対応する経路画像を表示する表示部と、
前記経路の位置と前記位置演算部によって求められた前記作業機の位置とに基づき、所定の位置を視点として前記作業機によって遮蔽される前記経路の部分に対応する前記経路画像を、前記作業機によって遮蔽されない前記経路の部分に対応する前記経路画像とは異なる形態で前記表示部に表示させる表示処理部と、
を含む、作業機械の画像表示システム。 - 前記表示処理部は、
前記経路画像を、前記所定の位置から見た画像に変換して、前記表示部に表示させる、請求項1に記載の作業機械の画像表示システム。 - 前記作業機械に取り付けられて対象を撮像する撮像装置を有し、
前記表示処理部は、前記経路画像と、前記撮像装置によって撮像された前記作業機の画像とを合成して、前記表示部に表示させる、請求項2に記載の作業機械の画像表示システム。 - 前記所定の位置は、前記作業機械に取り付けられて対象を撮像する撮像装置の位置である、請求項3に記載の作業機械の画像表示システム。
- 前記表示処理部は、
前記作業機と前記経路との間の距離に応じて、前記経路画像の表示形態を変更する、請求項1から請求項4のいずれか1項に記載の作業機械の画像表示システム。 - 請求項1から請求項5のいずれか1項に記載の作業機械の画像表示システムと、
前記作業機を操作する操作装置と、
を含む、作業機械の遠隔操作システム。 - 請求項1から請求項5のいずれか1項に記載の作業機械の画像表示システムを備えた、作業機械。
- 作業機械の作業機の位置を求め、
前記作業機が移動するであろう経路の位置と前記作業機の位置とに基づいて、所定の位置を視点として前記作業機が前記経路を遮蔽するか否かを判定し、
前記作業機によって遮蔽される前記経路の部分に対応する経路画像を、前記経路が前記作業機によって遮蔽されない前記経路の部分に対応する前記経路画像とは異なる形態で表示部に表示させる、
作業機械の画像表示方法。
Priority Applications (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US16/316,093 US20210285184A1 (en) | 2016-08-31 | 2017-08-24 | Image display system of work machine, remote operation system of work machine, work machine, and method for displaying image of work machine |
CA3029812A CA3029812C (en) | 2016-08-31 | 2017-08-24 | Image display system of work machine, remote operation system of work machine, work machine, and method for displaying image of work machine |
AU2017318911A AU2017318911B2 (en) | 2016-08-31 | 2017-08-24 | Image display system of work machine, remote operation system of work machine, work machine, and method for displaying image of work machine |
JP2018537209A JP6832548B2 (ja) | 2016-08-31 | 2017-08-24 | 作業機械の画像表示システム、作業機械の遠隔操作システム、作業機械及び作業機械の画像表示方法 |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2016170343 | 2016-08-31 | ||
JP2016-170343 | 2016-08-31 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2018043299A1 true WO2018043299A1 (ja) | 2018-03-08 |
Family
ID=61300788
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/JP2017/030406 WO2018043299A1 (ja) | 2016-08-31 | 2017-08-24 | 作業機械の画像表示システム、作業機械の遠隔操作システム、作業機械及び作業機械の画像表示方法 |
Country Status (5)
Country | Link |
---|---|
US (1) | US20210285184A1 (ja) |
JP (1) | JP6832548B2 (ja) |
AU (1) | AU2017318911B2 (ja) |
CA (1) | CA3029812C (ja) |
WO (1) | WO2018043299A1 (ja) |
Cited By (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2019172409A (ja) * | 2018-03-27 | 2019-10-10 | 株式会社タダノ | 作業車両 |
WO2019238328A1 (de) * | 2018-06-12 | 2019-12-19 | Robert Bosch Gmbh | Verfahren zur visualisierung |
WO2020003632A1 (ja) * | 2018-06-29 | 2020-01-02 | 株式会社小松製作所 | 表示制御装置、表示制御システムおよび表示制御方法 |
WO2020003631A1 (ja) * | 2018-06-29 | 2020-01-02 | 株式会社小松製作所 | 表示制御装置、および表示制御方法 |
WO2021020292A1 (ja) * | 2019-07-26 | 2021-02-04 | 株式会社小松製作所 | 表示システム、遠隔操作システム、及び表示方法 |
WO2021241716A1 (ja) * | 2020-05-27 | 2021-12-02 | 住友重機械工業株式会社 | ショベル用の施工支援システム |
US11434623B2 (en) * | 2018-09-25 | 2022-09-06 | Hitachi Construction Machinery Co., Ltd. | Work-implement external-shape measurement system, work-implement external-shape display system, work-implement control system and work machine |
Families Citing this family (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP7165638B2 (ja) * | 2019-09-25 | 2022-11-04 | 日立建機株式会社 | 作業機械 |
KR20210089039A (ko) * | 2020-01-07 | 2021-07-15 | 두산인프라코어 주식회사 | 건설기계의 제어 시스템 및 방법 |
Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5996702A (en) * | 1995-06-29 | 1999-12-07 | David Hall | System for monitoring movement of a vehicle tool |
JP2000116947A (ja) * | 1998-10-20 | 2000-04-25 | Taito Corp | 課題解決型乗り物ゲーム装置 |
JP2002146846A (ja) * | 2000-11-13 | 2002-05-22 | Topcon Corp | 建設機械の施工案内装置 |
JP2005114235A (ja) * | 2003-10-07 | 2005-04-28 | Hitachi Constr Mach Co Ltd | 移動式作業機の埋設物処理支援装置 |
JP2006053922A (ja) * | 2004-08-10 | 2006-02-23 | Caterpillar Inc | 機械操作者にとっての視認性を高める方法及び装置 |
JP2016061054A (ja) * | 2014-09-17 | 2016-04-25 | 住友重機械工業株式会社 | ショベル及びショベルの支援装置 |
Family Cites Families (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2004348095A (ja) * | 2003-03-26 | 2004-12-09 | National Institute Of Advanced Industrial & Technology | トレーニングシステム |
GB0410415D0 (en) * | 2004-05-11 | 2004-06-16 | Bamford Excavators Ltd | Operator display system |
US20060178820A1 (en) * | 2005-02-04 | 2006-08-10 | Novariant, Inc. | System and method for guiding an agricultural vehicle through a recorded template of guide paths |
US8290204B2 (en) * | 2008-02-12 | 2012-10-16 | Certusview Technologies, Llc | Searchable electronic records of underground facility locate marking operations |
CN103180522B (zh) * | 2010-10-22 | 2016-01-13 | 日立建机株式会社 | 作业机械的周边监视装置 |
US10610172B2 (en) * | 2012-07-17 | 2020-04-07 | Koninklijke Philips N.V. | Imaging system and method for enabling instrument guidance |
CN107343381A (zh) * | 2016-03-01 | 2017-11-10 | 株式会社小松制作所 | 评价装置以及评价方法 |
-
2017
- 2017-08-24 WO PCT/JP2017/030406 patent/WO2018043299A1/ja active Application Filing
- 2017-08-24 JP JP2018537209A patent/JP6832548B2/ja active Active
- 2017-08-24 AU AU2017318911A patent/AU2017318911B2/en active Active
- 2017-08-24 US US16/316,093 patent/US20210285184A1/en active Pending
- 2017-08-24 CA CA3029812A patent/CA3029812C/en active Active
Patent Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5996702A (en) * | 1995-06-29 | 1999-12-07 | David Hall | System for monitoring movement of a vehicle tool |
JP2000116947A (ja) * | 1998-10-20 | 2000-04-25 | Taito Corp | 課題解決型乗り物ゲーム装置 |
JP2002146846A (ja) * | 2000-11-13 | 2002-05-22 | Topcon Corp | 建設機械の施工案内装置 |
JP2005114235A (ja) * | 2003-10-07 | 2005-04-28 | Hitachi Constr Mach Co Ltd | 移動式作業機の埋設物処理支援装置 |
JP2006053922A (ja) * | 2004-08-10 | 2006-02-23 | Caterpillar Inc | 機械操作者にとっての視認性を高める方法及び装置 |
JP2016061054A (ja) * | 2014-09-17 | 2016-04-25 | 住友重機械工業株式会社 | ショベル及びショベルの支援装置 |
Cited By (15)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP7106929B2 (ja) | 2018-03-27 | 2022-07-27 | 株式会社タダノ | 作業車両 |
JP2019172409A (ja) * | 2018-03-27 | 2019-10-10 | 株式会社タダノ | 作業車両 |
WO2019238328A1 (de) * | 2018-06-12 | 2019-12-19 | Robert Bosch Gmbh | Verfahren zur visualisierung |
US11427988B2 (en) | 2018-06-29 | 2022-08-30 | Komatsu Ltd. | Display control device and display control method |
JP2020002718A (ja) * | 2018-06-29 | 2020-01-09 | 株式会社小松製作所 | 表示制御装置、および表示制御方法 |
JP2020002717A (ja) * | 2018-06-29 | 2020-01-09 | 株式会社小松製作所 | 表示制御装置、表示制御システムおよび表示制御方法 |
WO2020003631A1 (ja) * | 2018-06-29 | 2020-01-02 | 株式会社小松製作所 | 表示制御装置、および表示制御方法 |
WO2020003632A1 (ja) * | 2018-06-29 | 2020-01-02 | 株式会社小松製作所 | 表示制御装置、表示制御システムおよび表示制御方法 |
JP7285051B2 (ja) | 2018-06-29 | 2023-06-01 | 株式会社小松製作所 | 表示制御装置、および表示制御方法 |
JP7372029B2 (ja) | 2018-06-29 | 2023-10-31 | 株式会社小松製作所 | 表示制御装置、表示制御システムおよび表示制御方法 |
US11434623B2 (en) * | 2018-09-25 | 2022-09-06 | Hitachi Construction Machinery Co., Ltd. | Work-implement external-shape measurement system, work-implement external-shape display system, work-implement control system and work machine |
WO2021020292A1 (ja) * | 2019-07-26 | 2021-02-04 | 株式会社小松製作所 | 表示システム、遠隔操作システム、及び表示方法 |
JP7351478B2 (ja) | 2019-07-26 | 2023-09-27 | 株式会社小松製作所 | 表示システム、遠隔操作システム、及び表示方法 |
US11939744B2 (en) | 2019-07-26 | 2024-03-26 | Komatsu Ltd. | Display system, remote operation system, and display method |
WO2021241716A1 (ja) * | 2020-05-27 | 2021-12-02 | 住友重機械工業株式会社 | ショベル用の施工支援システム |
Also Published As
Publication number | Publication date |
---|---|
JP6832548B2 (ja) | 2021-02-24 |
JPWO2018043299A1 (ja) | 2019-06-24 |
AU2017318911A1 (en) | 2019-01-24 |
CA3029812C (en) | 2020-07-14 |
CA3029812A1 (en) | 2018-03-08 |
US20210285184A1 (en) | 2021-09-16 |
AU2017318911B2 (en) | 2020-07-02 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
JP6832548B2 (ja) | 作業機械の画像表示システム、作業機械の遠隔操作システム、作業機械及び作業機械の画像表示方法 | |
US11384515B2 (en) | Image display system for work machine, remote operation system for work machine, and work machine | |
US11634890B2 (en) | Image display system for work machine | |
JP7420733B2 (ja) | 表示制御システムおよび表示制御方法 | |
JP2024028438A (ja) | 作業機械の画像表示システム及び作業機械の画像表示方法 | |
JP2024052764A (ja) | 表示制御装置及び表示方法 | |
JP7128497B2 (ja) | 作業機械の画像表示システム | |
JP7333551B2 (ja) | 作業機械の画像表示システム | |
US11908076B2 (en) | Display system and display method | |
WO2022070707A1 (ja) | 表示制御装置及び表示方法 | |
JP6923144B2 (ja) | 作業機械の画像表示システム | |
US20210395980A1 (en) | System and method for work machine |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
WWE | Wipo information: entry into national phase |
Ref document number: 2018537209 Country of ref document: JP |
|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 17846304 Country of ref document: EP Kind code of ref document: A1 |
|
ENP | Entry into the national phase |
Ref document number: 3029812 Country of ref document: CA |
|
ENP | Entry into the national phase |
Ref document number: 2017318911 Country of ref document: AU Date of ref document: 20170824 Kind code of ref document: A |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 17846304 Country of ref document: EP Kind code of ref document: A1 |