JP2015226094A - Remote control system for work machine - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a remote control system which allows an operator to see what he/she wants in a more intuitive method through an imaging apparatus attached to a shovel to be remotely controlled.SOLUTION: A remote control system 100 comprises: a remote control device 50 to be used for remote control of a shovel; an imaging apparatus 41 attached to the shovel; a head-mounted display 51 which can display an image captured by the imaging apparatus 41; a wearable sensor 52 which detects movement of the head of a shovel operator; and a controller 30 which is mounted on the shovel and can communicate with the remote control device 50, the head-mounted display 51, and the wearable sensor 52. The controller 30, when the wearable sensor 52 detects a predetermined movement of the head of the operator, changes content of the image which is displayed in the head-mounted display 51 in accordance with the predetermined movement.

Description

  The present invention relates to a work machine remote control system.

  A system for remotely operating a hydraulic excavator using a remote operation device is known (see Patent Document 1). In this system, an operator remotely operates the hydraulic excavator while viewing an image of a television camera mounted on the hydraulic excavator. The TV camera is attached to a hydraulic excavator via a pan head. Therefore, the operator can adjust the panning angle of the television camera by remotely operating the camera platform using the camera platform operating device in the remote operation device. Further, this system automatically adjusts the panning angle of the TV camera according to the operation direction and the operation amount of the turning operation means when the turning operation means in the remote control device is operated, and the operation direction and The upper swing body is swung according to the operation amount. With this configuration, the operator can visually recognize the front in the turning direction when the upper turning body turns without the operator having to operate the pan head and the turning operation means at the same time.

JP-A-8-20975

  However, in the above-described system, the operator changes the panning angle of the TV camera unless remote control of the pan head via the pan head operating device or automatic adjustment of the panning angle according to the operation of the turning operation means is used. Can not. For this reason, the user cannot see what he wants to see without complicated operations.

  A remote control system for a work machine according to an embodiment of the present invention includes a remote control device used for remote control of a work machine, an imaging device attached to the work machine, and a head capable of displaying an image captured by the imaging device. A mount display, a wearable sensor for detecting a movement of a part of a body of an operator of the work machine, and a control mounted on the work machine and capable of communicating with the remote control device, the head mounted display, and the wearable sensor And the control device, when the wearable sensor detects a predetermined movement of a part of the operator's body, an image displayed on the head-mounted display according to the predetermined movement. Change the contents.

  With the above-described means, there is provided a remote control system for a work machine that enables an operator to see a place desired to be viewed in a more intuitive manner through an imaging device attached to the remote operation target work machine.

It is a side view of the shovel in which the remote control system which concerns on the Example of this invention is mounted. It is a block diagram which shows the structural example of a remote control system. It is a figure which shows the structural example of a movable pan head. It is a figure which shows the structural example of a remote control apparatus. It is a flowchart which shows an example of the flow of a transmission image control process. It is a figure which shows the example of the image which a head mounted display displays. It is a block diagram which shows another structural example of a remote control system.

  FIG. 1 is a side view of an excavator as a work machine on which a remote control system 100 according to an embodiment of the present invention is mounted. The excavator in FIG. 1 functions as an unmanned operating device that is remotely operated using a portable remote control device 50. However, the excavator of FIG. 1 can also function as a manned operating machine on which an operator gets on and operates. Further, an upper swing body 3 is mounted on the lower traveling body 1 of the excavator via a swing mechanism 2. A boom 4 is attached to the upper swing body 3. An arm 5 is attached to the tip of the boom 4, and a bucket 6 is attached to the tip of the arm 5. The boom 4, the arm 5, and the bucket 6 constitute a digging attachment, and are hydraulically driven by the boom cylinder 7, the arm cylinder 8, and the bucket cylinder 9, respectively. Further, the upper swing body 3 is provided with a cabin 10 and is mounted with a power source such as an engine. A controller 30 is installed in the cabin 10, and an imaging device 41 is attached via a movable camera platform 42. A radio communication device 40 is attached to the upper part of the upper swing body 3. Then, the operator remotely operates the excavator using the remote operation device 50 while wearing the head mounted display 51 and the wearable sensor 52 on the head and visually recognizing the image captured by the imaging device 41.

  FIG. 2 is a block diagram illustrating a configuration example of the remote operation system 100. The remote operation system 100 mainly includes a controller 30, a work machine control device 35, a wireless communication device 40, an imaging device 41, a movable camera platform 42, a remote operation device 50, a head mounted display 51, and a wearable sensor 52.

  The controller 30 is a control device that performs drive control of the shovel. In this embodiment, the controller 30 is composed of an arithmetic processing unit including a CPU and an internal memory. The controller 30 implements various functions by causing the CPU to execute programs corresponding to various functional elements such as the work machine control unit 31 and the transmission image control unit 32.

  The work machine control device 35 is a device used to control the work machine. In this embodiment, the work machine control device 35 is a device that operates in response to a control command from the controller 30 and includes a solenoid valve, an inverter, and the like. The solenoid valve is used to control the flow of hydraulic oil in a hydraulic circuit including a hydraulic actuator, a hydraulic pump, a pump regulator, and the like. The hydraulic actuator includes a boom cylinder 7, an arm cylinder 8, a bucket cylinder 9, and the like. Specifically, the electromagnetic valve is used, for example, to adjust the pilot pressure of a control valve that controls the flow rate of hydraulic oil supplied to the boom cylinder 7. The inverter is used to control the current flowing through the engine assist motor generator, the turning motor, and the like.

  The wireless communication device 40 is a device that controls wireless communication between the work machine and an external device. In the present embodiment, the wireless communication device 40 controls wireless communication between the excavator and each of the remote control device 50, the head mounted display 51, and the wearable sensor 52 using a specific low power wireless in the 429 MHz band. The wireless communication device 40 may use a wireless communication standard such as Wi-Fi (registered trademark), a short-range wireless communication standard such as Bluetooth (registered trademark), or the like.

  The imaging device 41 is a device that is attached to the work machine and captures an image around the work machine. In the present embodiment, the imaging device 41 is a camera using an imaging element such as a CMOS or a CCD, and outputs a captured image to the controller 30.

  The movable head 42 is a device for changing the imaging direction of the imaging device 41. In the present embodiment, the movable head 42 is an electric head that changes the imaging direction of the imaging device 41 in accordance with a control command from the controller 30.

  FIG. 3 shows a configuration example of the movable head 42. The movable head 42 mainly includes a tilt motor 42a, a bracket 42b, and a pan motor 42c. The tilt motor 42 a is an electric motor that rotates the imaging device 41 around the horizontal axis in accordance with a control command from the controller 30. In the present embodiment, the tilt motor 42a has a rotation shaft coupled to a tilt shaft 42aX extending in the horizontal direction. The tilt shaft 42aX is fixed to the imaging device 41 so as to rotate integrally with the imaging device 41. The bracket 42b is a member that supports the imaging device 41, the tilt motor 42a, and the tilt shaft 42aX. The pan motor 42 c is an electric motor that rotates the imaging device 41 around the vertical axis in response to a control command from the controller 30. In the present embodiment, the pan motor 42c has a rotation shaft coupled to the pan shaft 42cX extending in the vertical direction. The pan shaft 42cX is fixed to the bracket 42b so as to rotate integrally with the bracket 42b. With this configuration, the movable head 42 can direct the imaging direction of the imaging device 41 in an arbitrary direction.

  The remote operation device 50 is a device used for remote operation of the work machine. In this embodiment, the remote operation device 50 is a device having a transmission function, and the controller 30 transmits remote operation information necessary for controlling the work machine control device 35 to the wireless communication device 40.

  FIG. 4 shows a configuration example of the remote operation device 50. The remote operation device 50 mainly includes a left operation lever 50a, a central operation lever 50b, a right operation lever 50c, a stop button 50d, and a shoulder belt 50e. The operation levers 50a, 50b, and 50c are all four-way joysticks that return to the neutral position when the operator releases the operation lever.

  In addition, the operation levers 50a, 50b, and 50c have buttons 50a1, 50b1, and 50c1 at their tips, respectively. When the operation levers 50a, 50b, and 50c are operated without pressing the buttons 50a1, 50b1, and 50c1, the remote operation device 50 does not output information according to the operation. This is to prevent the shovel from operating when the operator accidentally touches the operation levers 50a, 50b, and 50c.

  Specifically, the operator opens and closes the arm 5 by tilting the left operation lever 50a back and forth while pressing the button 50a1, and the upper swing body by tilting the left operation lever 50a left and right while pressing the button 50a1. 3 is turned left and right. Further, the operator causes the excavator to travel forward, backward, left, and right by tilting the central operation lever 50b forward, backward, left and right while pressing the button 50b1. Further, the operator raises and lowers the boom 4 by tilting the right operation lever 50c back and forth while pressing the button 50c1, and opens and closes the bucket 6 by tilting the right operation lever 50c left and right while pressing the button 50c1. Further, the operator stops the excavator by pressing the stop button 50d.

  The head mounted display 51 is a display device attached to the head. In the present embodiment, the head mounted display 51 employs a monocular non-transmissive display positioned on the right eye of the operator. The head mounted display 51 has a receiving function, and receives and displays image information transmitted from the excavator through the wireless communication device 40. With this configuration, for example, the operator can visually recognize the excavation area in front of the shovel with the right eye while visually confirming the movement of the entire shovel from the periphery of the shovel with the left eye. However, the head mounted display 51 may employ a binocular transmissive display.

  In addition, the head mounted display 51 can move the non-transmissive display positioned in front of the operator's right eye to a retracted position that does not interfere with the visual field of the operator's right eye. Therefore, when the operator moves around the shovel, the operator may move the non-transmissive display to the retracted position to ensure the right eye field of view. The same applies to a binocular transmissive display.

  The wearable sensor 52 is a sensor that can be worn by the operator and can detect movement of a part of the operator's body. In this embodiment, the wearable sensor 52 is an acceleration sensor that is integrally attached to the head mounted display 51, and detects a predetermined movement of the operator's head. The wearable sensor 52 has a transmission function, and transmits the detection result to the wireless communication device 40. The wearable sensor 52 may transmit an inclination angle with respect to a predetermined reference posture of the operator's head toward the wireless communication device 40 as a predetermined movement of the operator's head.

  The wearable sensor 52 may be a sensor that detects the movement of the eyes of the operator. Specifically, the wearable sensor 52 may be an image sensor that is attached to the head mounted display 51 and images an operator's pupil.

  Next, the work machine control unit 31 and the transmission image control unit 32 that are functional elements of the controller 30 will be described.

  The work machine control unit 31 is a functional element that operates the work machine by controlling the work machine control device 35 in accordance with operation information from an operation device such as an operation lever in the work machine or remote operation information from a remote operation device. is there. The work machine control unit 31 may operate the work machine by controlling the work machine control device 35 exclusively in accordance with remote operation information from the remote operation device. In the present embodiment, the work machine control unit 31 controls the work machine control device 35 according to the remote operation information from the remote operation device 50 to operate the shovel. For example, when the operator performs an operation of opening the arm 5 by tilting the left operation lever 50a forward while pressing the button 50a1 with the remote operation device 50, the remote operation device 50 wirelessly communicates information on the arm opening operation command. Transmit to the device 40. When the work machine control unit 31 of the controller 30 receives information related to the arm opening operation command via the wireless communication device 40, the work machine control unit 31 outputs the control command to an electromagnetic valve capable of adjusting the pilot pressure of the control valve related to the arm cylinder 8. . The electromagnetic valve generates a pilot pressure according to the control command and operates the control valve so that the hydraulic oil discharged from the hydraulic pump is supplied to the rod side oil chamber of the arm cylinder 8.

  The transmission image control unit 32 is a functional element that controls the content of an image transmitted to the head mounted display 51 through the wireless communication device 40. In the present embodiment, when the wearable sensor 52 determines that the wearable sensor 52 has detected a predetermined movement of the operator's head, the transmission image control unit 32 sends a control command corresponding to the predetermined movement to the movable head 42. Output. The predetermined movement of the head is the vertical rotation of the head about the left and right axis (pitch down, pitch up), the left and right rotation of the head about the vertical axis (right yaw, left yaw), the left and right of the head about the front and rear axis Includes rotation (right roll, left roll), etc. The movable head 42 changes the imaging direction of the imaging device 41 according to the control command. The imaging device 41 continuously outputs captured images to the controller 30 regardless of whether or not the imaging direction is changed. As a result, the transmission image control unit 32 can continuously transmit the captured images of the imaging device 41 having different imaging directions before and after the predetermined movement of the operator's head toward the head mounted display 51.

  Specifically, when it is determined that the wearable sensor 52 has detected the right yaw of the operator's head, the transmission image control unit 32 transmits a right yaw command corresponding to the right yaw to the movable head 42. . In response to the right yaw command, the movable head 42 pans (shifts to the right) the imaging direction of the imaging device 41 to the right by a predetermined angle. The imaging device 41 outputs to the controller 30 an image captured in the imaging direction panned to the right by a predetermined angle. As a result, the transmission image control unit 32 transmits the captured image of the imaging device 41 whose imaging direction is panned to the right by a predetermined angle toward the head mounted display 51.

  Further, the transmission image control unit 32 may control the movable camera platform 42 according to the output of the wearable sensor 52 so that the orientation of the operator's head and the orientation of the imaging device 41 are synchronized.

  Alternatively, when it is determined that the wearable sensor 52 has detected the pitch down of the operator's head, the transmission image control unit 32 directs another image that is not a captured image of the imaging device 41 toward the head mounted display 51. You may send it. In this case, the transmission image control unit 32 may transmit, for example, an image of an instrument screen indicating a state such as a remaining fuel amount, a hydraulic oil temperature, a cooling water temperature, and an accumulated operation time toward the head mounted display 51. The instrument screen may be the same screen as the instrument screen displayed on the vehicle-mounted display or the like installed in the excavator cabin 10 or may be a different screen. Thereafter, when it is determined that the wearable sensor 52 has detected the pitch up of the operator's head, the transmission image control unit 32 stops the transmission of the instrument screen image and restarts the transmission of the captured image of the imaging device 41. May be.

  Alternatively, the transmission image control unit 32 may increase or decrease the zoom magnification of the imaging device 41 when it is determined that the wearable sensor 52 has detected the right roll / left roll of the operator's head.

  The transmission image control unit 32 may accept the detection result of the wearable sensor 52 and change the imaging direction of the imaging device 41 only when a predetermined condition is satisfied. For example, the transmission image control unit 32 may accept the detection result of the wearable sensor 52 only when an operation detection switch (not shown) in the remote operation device 50 is switched on. This is to prevent the imaging direction of the imaging device 41 from being changed against the operator's will. Note that the transmission image control unit 32 may accept a detection result of the wearable sensor 52 after that when the wearable sensor 52 detects a specific movement of a part of the operator's body.

  Next, with reference to FIG. 5, a process for controlling the content of an image transmitted by the controller 30 toward the head mounted display 51 (hereinafter referred to as “transmission image control process”) will be described. FIG. 5 is a flowchart showing an example of the flow of transmission image control processing. The controller 30 repeatedly executes this transmission image control process at a predetermined control cycle.

  First, the transmission image control unit 32 of the controller 30 determines whether the operation detection function is valid (step S1). In the present embodiment, the transmission image control unit 32 determines whether the operation detection switch of the remote operation device 50 is turned on based on the remote operation information from the remote operation device 50 received through the wireless communication device 40.

  When it is determined that the motion detection function is not valid (NO in step S1), the transmission image control unit 32 ends the current transmission image control process. In this embodiment, when it is detected that the operation detection switch is off, the transmission image control unit 32 ends the current transmission image control process.

  On the other hand, if it is determined that the motion detection function is valid (YES in step S1), the transmission image control unit 32 determines whether a predetermined movement of a part of the operator's body has been detected (step S2). In this embodiment, the transmission image control unit 32 determines whether a predetermined movement of the operator's head has been detected based on the detection result of the wearable sensor 52 received through the wireless communication device 40.

  And when it determines with the predetermined motion of a part of operator's body not being detected (NO of step S2), the transmission image control part 32 complete | finishes this transmission image control process.

  On the other hand, when it is determined that a predetermined movement of a part of the operator's body has been detected (YES in step S2), the transmission image control unit 32 directs the image corresponding to the predetermined movement toward the head mounted display 51. Transmit (step S3). In this embodiment, when the transmission image control unit 32 determines that a predetermined movement of the operator's head is detected by the wearable sensor 52, the transmission direction of the imaging device 41 in the reference posture is determined according to the predetermined movement. Change. Note that the reference posture of the imaging device 41 is, for example, a state in which the imaging device 41 faces the front of the excavator in a horizontal posture.

  Specifically, when the wearable sensor 52 determines that the right yaw of the operator's head has been detected by the wearable sensor 52, the transmission image control unit 32 controls the movable camera platform 42 to capture the imaging direction of the imaging device 41 in the reference posture. May be changed diagonally downward to the right. In this case, when the transmission image control unit 32 determines that the wearable sensor 52 has detected the left yaw of the operator's head, the transmission image control unit 32 controls the movable camera platform 42 to set the imaging device 41 facing obliquely downward to the right to the reference posture. It may be restored. Further, when it is determined that the wearable sensor 52 has detected the pitch down of the operator's head, the transmission image control unit 32 transmits an instrument screen image to the head mounted display 51 instead of the captured image of the imaging device 41. Also good. In this case, when it is determined that the wearable sensor 52 has detected the pitch up of the operator's head, the transmission image control unit 32 stops transmission of the instrument screen image and restarts transmission of the captured image of the imaging device 41. May be.

  FIG. 6 is a diagram illustrating an example of an image displayed by the head mounted display 51. Specifically, FIG. 6A is an example of an image captured by the imaging device 41 in the reference posture. FIG. 6B is an example of an image captured by the image capturing device 41 when the image capturing direction of the image capturing device 41 in the reference posture is changed diagonally downward to the right. FIG. 6C is an example of an instrument screen. It is an example.

  In the above example, the head mounted display 51 that has displayed the image of FIG. 6A displays the image of FIG. 6B when the right yaw of the operator's head is detected. When the left yaw of the operator's head is subsequently detected, the image in FIG. 6A is displayed again. Further, the head mounted display 51 that has displayed the image of FIG. 6A displays the image of FIG. 6C when the pitch down of the operator's head is detected. Then, when the pitch up of the operator's head is subsequently detected, the image in FIG. 6A is displayed again.

  With the remote operation system 100 having the above configuration, the operator can change the imaging direction of the imaging device 41 to a desired direction without using a hand. Further, the operator can switch the captured image of the imaging device 41 to a predetermined image such as an instrument screen image without using a hand. Therefore, the operator can display a desired image on the head mounted display 51 while operating the remote control device 50 with both hands.

  Next, a remote operation system 100A, which is another configuration example of the remote operation system, will be described with reference to FIG. FIG. 7 is a block diagram illustrating a configuration example of the remote operation system 100A, and corresponds to FIG. 7 is different from the remote operation system 100 of FIG. 2 in that the remote operation system 100A of FIG. 7 includes three imaging devices 41A to 41C and the movable camera platform 42 is omitted, but is common in other points. . Therefore, description of common parts is omitted, and different parts are described in detail.

  The imaging devices 41A to 41C are imaging devices that are attached to a shovel with the imaging direction fixed. Specifically, the first imaging device 41A is attached to the front center portion of the upper surface of the cabin 10 so as to image the front of the excavator obliquely from above. Further, the second imaging device 41B is attached to the front right corner of the upper surface of the cabin 10 so as to image the right diagonal front of the shovel from diagonally above. Further, the third imaging device 41C is attached to the front left corner of the upper surface of the cabin 10 so as to capture the left oblique front of the excavator from obliquely above.

  When it is determined that the wearable sensor 52 has detected a predetermined movement of the operator's head, the transmission image control unit 32 picks up any one of the three imaging devices 41A to 41C according to the predetermined movement. To be transmitted to the head mounted display 51.

  Specifically, when it is determined that the wearable sensor 52 has detected the right yaw of the operator's head, the transmission image control unit 32 outputs the second captured image of the first imaging device 41A that has been transmitted so far. Switch to the captured image of the imaging device 41B. Thereafter, when it is determined that the wearable sensor 52 has detected the left yaw of the operator's head, the transmission image control unit 32 stops the transmission of the captured image of the second imaging device 41B and the first imaging device 41A. Resume the transmission of the captured image.

  Similarly, if the wearable sensor 52 can determine that the wearable sensor 52 has detected the left yaw of the operator's head, the transmission image control unit 32 uses the first imaging device 41A that has been transmitted so far as the third imaging device. Switch to the captured image of 41C. Thereafter, when it is determined that the wearable sensor 52 has detected the right yaw of the operator's head, the transmission image control unit 32 stops transmission of the captured image of the third imaging device 41C and the first imaging device 41A. Resume the transmission of the captured image.

  As in the case of the remote operation system 100, the transmission image control unit 32 directs the instrument screen image to the head mounted display 51 when it is determined that the wearable sensor 52 has detected the pitch down of the operator's head. You may send it. In this case, the transmission image control unit 32 stops the transmission of the instrument screen image when the wearable sensor 52 can determine that the pitch up of the operator's head has been detected, and resumes the transmission of the captured image of the imaging device 41. You may let them.

  Similarly to the remote operation system 100, the transmission image control unit 32 increases / decreases the display magnification of the captured image when the wearable sensor 52 can determine that the right roll / left roll of the operator's head has been detected. May be.

  With the remote operation system 100A having the above configuration, the operator can display a desired image on the head mounted display 51 while operating the remote operation device 50 with both hands, as in the case of the remote operation system 100. .

  Note that the remote operation system 100A may be configured to switch the captured images of four or more imaging devices according to the detection result of the wearable sensor 52 and transmit the images to the head mounted display 51, and the imaging of the two imaging devices. The image may be switched and transmitted to the head mounted display 51.

  Further, since the remote operation system 100A omits the movable pan head 42, the fault tolerance can be improved as compared with the remote operation system 100.

  Although the preferred embodiments of the present invention have been described in detail above, the present invention is not limited to the above-described embodiments, and various modifications and substitutions can be made to the above-described embodiments without departing from the scope of the present invention. Can be added.

  For example, in the above-described embodiment, the remote control device 50, the head mounted display 51, and the wearable sensor 52 perform wireless communication with the wireless communication device 40 independently of each other. However, the present invention is not limited to this configuration. For example, the head mounted display 51 and the wearable sensor 52 may perform wireless communication with the wireless communication device 40 through a common transceiver attached to the head mounted display 51. The head mounted display 51 and the wearable sensor 52 perform wired communication or wireless communication with the remote operation device 50 and perform wireless communication with the wireless communication device 40 through a common transceiver attached to the remote operation device 50. You may do it.

  In the above-described embodiments, the remote control systems 100 and 100A are applied to excavators, but may be applied to other work machines such as asphalt finishers and cranes.

  DESCRIPTION OF SYMBOLS 1 ... Lower traveling body 2 ... Turning mechanism 3 ... Upper turning body 4 ... Boom 5 ... Arm 6 ... Bucket 7 ... Boom cylinder 8 ... Arm cylinder 9 ... Bucket cylinder 10 ... cabin 30 ... controller 31 ... work machine control unit 32 ... transmission image control unit 35 ... work machine control device 40 ... wireless communication device 41, 41A, 41B, 41C ... Imaging device 42 ... Movable pan head 42a ... Tilt motor 42aX ... Tilt shaft 42b ... Bracket 42c ... Pan motor 42cX ... Pan shaft 50 ... Remote control device 50a, 50b, 50c ... Control lever 50d ... Stop button 50e ... Shoulder belt 51 ... Head mounted display 52 ... Wearer Rusensa 100,100A ··· remote control system

Claims (3)

A remote control device for remote control of the work machine;
An imaging device attached to the work machine;
A head mounted display capable of displaying an image captured by the imaging device;
A wearable sensor for detecting a movement of a part of an operator of the work machine;
A control device mounted on the work machine and capable of communicating with the remote control device, the head mounted display, and the wearable sensor;
When the wearable sensor detects a predetermined movement of a part of the operator's body, the control device changes the content of an image displayed on the head mounted display according to the predetermined movement.
Remote control system for work machines. The remote control device is portable,
The remote control system for work machines according to claim 1. The wearable sensor detects movement of the operator's head or eyes;
The remote control system for work machines according to claim 1 or 2.