JP6947101B2 - Remote control system and main control device - Google Patents

Description

The present invention relates to a remote control system or the like including a main operation device and a slave operation device.

In recent years, a remote control system that remotely controls a construction machine by using a main operation device called a master that directly receives an operator's operation and a slave operation device called a slave that is communicably connected to the master has been used. It is known (for example, Patent Document 1).

In such a remote control system, the surrounding image of the construction machine taken by the camera installed in the cockpit of the construction machine is displayed on the display device provided on the master, and the operator displays the surroundings displayed on the display device. Remotely operate the work equipment (boom, arm, bucket, etc.) of the construction machine while looking at the image.

As prior art documents related to the present application, there is Patent Document 2 in addition to Patent Document 1. Patent Document 2 detects the relative angle of the lower traveling body with respect to the upper turning body from the image taken by the rear surveillance camera provided at the rear of the construction machine, and provides an image showing the detected relative angle in the cockpit. Disclose the technology to display on the display device.

Japanese Unexamined Patent Publication No. 10-252101 Japanese Unexamined Patent Publication No. 2012-107395

However, depending on the possible posture of the work device, the work device may be located outside the field of view of the camera provided in the cockpit. In this case, the work device disappears from the surrounding image displayed on the display device of the master, and the operator of the master cannot confirm in which direction the work device is currently located, which causes anxiety to the operator.

In particular, when resuming remote control of a construction machine after a break such as a lunch break, if the work device does not appear in the surrounding image, the operator is required to carefully operate the work device while predicting the position of the work device. There is a problem that the operation load increases.

The above-mentioned Patent Document 2 is not an invention relating to a remote control system in the first place, and there is no disclosure about installing a camera in the cockpit. Therefore, when the work device disappears from the surrounding image displayed on the display screen of the master. The above-mentioned problems that arise cannot be solved.

An object of the present invention is to provide a remote control system or the like capable of causing an operator of the main operation device to recognize the position of the work device even if the work device disappears from the surrounding image displayed on the display device of the main operation device. That is.

One aspect of the present invention is a remote control system for remotely controlling a construction machine including a work device.
It is provided with a main operating device for remotely controlling the construction machine and a secondary operating device mounted on the construction machine and operating the working device based on an operation received by the main operating device.
The slave operation device is
A camera that acquires a surrounding image of the surroundings of the construction machine from inside the cockpit of the construction machine, and
A posture detection unit that detects the posture of the work device, and
It is provided with a first communication unit that transmits the surrounding image and the posture information indicating the posture of the work device detected by the posture detection unit to the main operation device.
The main operating device is
An operating member that accepts operator operations and
A second communication unit that receives the surrounding image and the posture information transmitted from the slave operation device, and
A display device that displays a display screen including the surrounding image received by the second communication unit, and
A determination unit that determines whether or not the work device is included in the surrounding image received by the second communication unit, and a determination unit.
When the determination unit determines that the surrounding image does not include the work device, the position of the work device is calculated from the posture information received by the second communication unit, and the calculated position is used as the operator. It is provided with a display control unit that displays a display object for notifying the display on the display screen.

According to this configuration, when the work device is not included in the surrounding image of the construction machine taken from the cockpit of the construction machine, the display object for notifying the operator of the position of the work device is the display screen of the main operation device. Is displayed in.

Therefore, even if the work device disappears from the surrounding image included in the display screen of the main operation device, the operator of the main operation device can recognize the position of the work device. As a result, the operator can operate the work device with peace of mind, and the work load of the operator can be reduced.

In the above configuration, it is preferable that the display control unit superimposes and displays at least a part of the display object on the surrounding image.

According to this configuration, at least a part of the display object is superimposed and displayed on the surrounding image, so that it is possible to avoid a situation in which the display object is not noticed by the operator.

In the above configuration, it is preferable that the display control unit stops the display of the display object when the determination unit determines that the work device is included in the surrounding image.

If the surrounding image contains a work device, the operator can recognize the position of the work device without displaying the display object. Further, in this case, when at least a part of the display object is superimposed and displayed on the surrounding image, a part of the surrounding image is hidden by the display object and the visibility is deteriorated.

According to this configuration, when the surrounding image includes a work device, the display object is not displayed, so that the deterioration of the visibility of the display screen can be suppressed and the operator can concentrate the remote control. can.

In the above configuration, it is preferable that the display control unit stops the display of the display object when the operation member is operated, regardless of the determination result by the determination unit.

Even if the work device disappears from the surrounding image during the operation of the operating member, the operator can predict the position of the work device to some extent from the current operation. According to this configuration, when the operating member is operated, the display object is not displayed regardless of the determination result by the determination unit, so that the amount of information displayed on the display device is suppressed and the operator can be remotely controlled. Can be concentrated.

In the above configuration, when the determination unit determines that the work device is not included in the surrounding image, the display control unit waits for a certain period of time or more after the operation member has not been operated. , It is preferable to display the display object.

According to this configuration, it is possible to prevent the occurrence of chattering in which the display and non-display of the display object are repeated in a short time in the scene where the appearance and disappearance of the work device are repeated in a short time with respect to the surrounding image. , The operator can concentrate the remote control of the construction machine.

In the above configuration, when the determination unit determines that the surrounding image does not include the work device at the start of communication between the main operation device and the slave operation device, the display control unit is the operation member. It is preferable to display the display object regardless of whether or not is operated.

According to this configuration, if the work device does not appear in the surrounding image at the start of communication, it is possible to prevent the operator from being unable to recognize the position of the work device.

In the above configuration, the main operating device further includes a shutoff lever that receives an operation from an operator for shutting off the operation of the operating member.
When the display control unit determines that the shutoff lever is locked and the determination unit does not include the work device in the surrounding image, the display control unit determines whether or not the operation member is operated. Regardless, it is preferable to display the display object.

According to this configuration, when the shutoff lever is locked, it is possible to prevent the operator from being unable to recognize the position of the work device when the work device does not appear in the surrounding image.

In the above configuration, the construction machine has a decelerator function that idles the engine of the construction machine when the operating member has not been operated for a certain period of time.
It is preferable that the display control unit displays the display object when the decelerator function is operating and the determination unit determines that the work device is not included in the surrounding image.

According to this configuration, it is possible to prevent the operator from being unable to recognize the position of the work device when the work device does not appear in the surrounding image while the decelerator function is operating.

In the above configuration, the display object is preferably composed of an arrow image that three-dimensionally indicates the direction in which the work device is located.

According to this configuration, the position of the working device can be three-dimensionally notified to the operator.

In the above configuration, it is preferable that the display object is composed of an arrow image that two-dimensionally indicates the direction in which the work device is located.

According to this configuration, the position of the working device can be two-dimensionally notified to the operator.

In the above configuration, the display object is preferably composed of a construction machine image simulating the actual posture of the construction machine from a side view.

According to this configuration, the position of the work device can be visually notified to the operator by using the construction machine image simulating the actual posture of the construction machine from the side view.

In the above configuration, it is preferable that the display object is composed of a work device image simulating the actual posture of the work device.

According to this configuration, the position of the work device can be visually notified to the operator by using the work device image simulating the actual posture of the work device.

In the above configuration, it is preferable that the display control unit displays the size of the arrow image larger as the position of the work device approaches the cockpit.

According to this configuration, the distance to the cockpit of the work device can be visually notified by the size of the arrow image.

In the above configuration, it is preferable that the display control unit directs the tip of the arrow image in the direction in which the work device is located with respect to the cockpit in the direction projected on the surrounding image.

According to this configuration, it is possible to accurately inform the operator who works while looking at the surrounding image of the direction of the arrow image.

In the above configuration, it is preferable that the display control unit displays the peripheral image on the display device so that the margin area is included adjacent to the peripheral image, and displays the display object in the margin area.

According to this configuration, since the display object is displayed in the margin area, it is possible to prevent the display object from interfering with the work of the operator.

According to the present invention, even if the working device disappears from the surrounding image displayed on the display device of the main operating device, the operator of the main operating device can recognize the position of the working device.

It is a block diagram which shows the whole structure of the remote control system which concerns on embodiment of this invention. It is a figure which shows the appearance of the construction machine shown in FIG. It is a figure which shows an example of the display screen displayed on the display device of a master. It is a figure which shows the pattern of the display position of a display object. It is a table which summarizes the display / non-display timing of the display object which concerns on this embodiment. It is a flowchart which shows an example of the processing of the remote control system which concerns on embodiment of this invention. It is a flowchart which shows an example of the process at the time of communication start of the remote control system which concerns on embodiment of this invention.

FIG. 1 is a block diagram showing an overall configuration of a remote control system according to an embodiment of the present invention. The remote control system includes a slave 10 and a master 20. The slave 10 is a slave operation device that is arranged in the cockpit of the construction machine 30 and directly operates the operation lever 31 of the construction machine 30 based on the operation amount received by the master 20. The slave 10 is, so to speak, a machine that operates the construction machine 30 as a dummy of the operator. The master 20 is a main operating device for directly receiving the operation of the operator of the construction machine 30 and remotely controlling the construction machine 30. In the present embodiment, the master 20 is composed of an operating device simulating the cockpit of the construction machine 30, and the operating lever 24 similar to the operating lever 31 of the construction machine 30 is arranged at the same position as the construction machine 30. ing. Further, the master 20 is provided with a seat on which the operator sits.

The slave 10 and the master 20 are connected to each other so as to be able to communicate with each other via the communication path 100. As the communication path 100, a specific power-saving radio and a communication path such as Bluetooth (registered trademark) for wirelessly communicating the slave 10 and the master 20 at a distance of about several tens of meters to several hundreds of meters are adopted. However, this is an example, and as the communication path 100, a public communication line including a mobile phone communication network, an Internet communication network, or the like may be adopted. In this case, the master 20 and the slave 10 can perform long-distance communication.

FIG. 2 is a diagram showing the appearance of the construction machine 30 shown in FIG. The construction machine 30 shown in FIG. 2 is composed of a hydraulic excavator. The construction machine 30 includes a crawler-type lower traveling body 310, an upper swivel body 320 rotatably provided on the lower traveling body 310, and a working device 330 attached to the upper swivel body 320.

The work device 330 swings with respect to the boom 331 undulatingly attached to the upper swing body 320, the arm 332 swingably attached to the tip of the boom 331, and the tip of the arm 332. It is equipped with an attachment 333 that is movably attached.

Further, the working device 330 swings the boom cylinder 334 that raises and lowers the boom 331 with respect to the upper swing body 320, the arm cylinder 335 that swings the arm 332 with respect to the boom 331, and the attachment 333 with respect to the arm 332. It is provided with an attachment cylinder 336 to be used. The upper swivel body 320 includes a driver's seat 3C on which the operator is boarded.

The reference is returned to FIG. The master 20 includes a communication unit 21 (an example of a second communication unit), a controller 22, a display device 23, an operation lever 24 (an example of an operation member), a shutoff lever 25, and sensors S1 and S2.

The communication unit 21 is composed of a communication device corresponding to the communication method adopted by the communication path 100, and receives the surrounding image of the construction machine 30 and the posture information indicating the posture of the work device 330 transmitted from the slave 10. .. The posture information of the work device 330 includes, for example, the detected value of the angle sensor 13. Further, the communication unit 21 transmits the operation amount received by the operation lever 24 of the master 20 to the slave 10.

The controller 22 is composed of a computer including a processor such as a CPU and an ASIC and a storage device such as a ROM and a RAM. The controller 22 includes a determination unit 221 and a display control unit 222. The determination unit 221 and the display control unit 222 may be realized by the processor executing the control program, or may be configured by a dedicated hardware circuit.

The determination unit 221 determines whether or not the work device 330 is included in the surrounding image received by the communication unit 21. Here, the determination unit 221 determines whether or not the work device 330 is included in the surrounding image by applying template matching using the template image of the work device 330 stored in the storage device in advance to the surrounding image. You may. Alternatively, the determination unit 221 stores in advance the detection value of the angle sensor 13 included in the posture information transmitted from the slave 10 and the determination result of whether or not the work device 330 is included in the surrounding image. By referring to the determination table stored in the apparatus, it may be determined whether or not the work apparatus 330 is included in the surrounding image.

The display control unit 222 causes the display device 23 to display the surrounding image received by the communication unit 21 from the slave 10. Further, when the determination unit 221 determines that the surrounding image does not include the work device 330, the display control unit 222 calculates the position of the work device 330 from the posture information received by the communication unit 21, and the calculated position. A display object for notifying the operator is displayed on the display device 23 together with the surrounding image.

The display device 23 is composed of a display device such as a liquid crystal display and an organic EL display provided in front of the seat of the master 20, and displays a surrounding image received from the slave 10 by the communication unit 21. Further, the display device 23 displays or does not display the display object under the control of the display control unit 222. The operator of the master 20 remotely controls the construction machine 30 while looking at the surrounding image displayed on the display device 23.

The operation lever 24 corresponds to the operation lever 31 included in the construction machine 30, and receives an operation from an operator who remotely controls the construction machine 30. In the present embodiment, the operation lever 24 will be described as an operation lever that receives an operator's operation for operating the work device 330 of the construction machine 30. For example, when the operating lever 24 is tilted forward, the boom 331 is tilted, when tilted backward, the boom 331 is undulated, and when tilted to the left, the attachment 333 is swung toward the driver's seat to the right. It may be composed of an ATT (attachment) lever that can tilt the attachment 333 in four directions, front, rear, left and right, which swings the attachment 333 from the driver's seat to the opposite side when tilted toward the driver's seat. Alternatively, the operating lever 24 may be composed of a traveling lever that moves the construction machine 30 forward or backward. Alternatively, the operating lever 24 may be composed of a swivel lever that can swing the arm 332 and tilt the upper swivel body 320 in four directions, front, back, left, and right. Alternatively, the operating lever 24 may be composed of three operating levers, an ATT lever, a traveling lever, and a swivel lever. In any case, the operation lever 24 includes various operation levers corresponding to the various operation levers included in the operation lever 31 of the construction machine 30.

The sensor S1 is composed of, for example, a potentiometer type sensor, detects the operation amount of the operation lever 24, and outputs the operation amount to the controller 22. Here, the sensor S1 individually detects each of the front, rear, left, and right operating amounts of the operating lever 24 and outputs the operating amount to the controller 22. The sensor S1 increases the amount of operation to be detected as the amount of inclination of the operation lever 24 increases in each of the front, rear, left, and right directions. When the operation lever 24 is composed of a plurality of operation levers, the sensor S1 is composed of a plurality of sensors corresponding to each operation lever.

The shutoff lever 25 is composed of an operation lever that can be tilted between the lock position and the release position, and receives an operation from an operator for shutting off the operation of the control lever 24. When the shutoff lever 25 is positioned at the locked position, the controller 22 keeps the posture of the working device 330 in that state, thereby shutting off the operation of the operating lever 24. On the other hand, when the shutoff lever 25 is positioned at the release position, the controller 22 releases the posture holding of the work device 330 and operates the work device 330 in response to the operation of the operation lever 24.

The sensor S2 is a sensor that detects whether the shutoff lever 25 is located at the locked position or the released position.

The slave 10 includes a communication unit 11 (an example of a first communication unit), a controller 12, an angle sensor 13 (an example of a posture detection unit), a camera 14, and an operation mechanism 15.

The communication unit 11 is composed of a communication device corresponding to the communication method adopted by the communication path 100, and transmits the surrounding image acquired by the camera 14 and the posture information of the work device 330 to the master 20. Further, the communication unit 11 receives the operation amount of the operation lever 24 received by the master 20 from the master 20.

The controller 12 is composed of a computer including a processor such as a CPU and an ASIC and a storage device such as a ROM and a RAM. When the communication connection between the master 20 and the slave 10 is established, the controller 12 periodically transmits the ambient image acquired by the camera 14 and the posture information of the work device 330 to the master 20 at a predetermined sampling cycle. The communication unit 11 is controlled so as to be.

The angle sensor 13 is composed of, for example, a resolver or potentiometer type sensor, and detects the posture of the working device 330. The angle sensor 13 includes a boom sensor that detects an angle (boom angle) of the boom 331 with respect to the upper swing body 320, an arm sensor that detects an angle (arm angle) of the arm 332 with respect to the boom 331, and an angle of the attachment 333 with respect to the arm 332. It is composed of an attachment sensor that detects (attachment angle) and a swivel sensor that detects the swivel angle of the upper swivel body 320 with respect to the lower traveling body 310.

Therefore, the display control unit 222 of the master 20 uses the detection values of the boom sensor, the arm sensor, and the attachment sensor and the known lengths of the boom 331, the arm 332, and the attachment 333 to position the tip of the attachment 333. Is calculated, and the calculated tip position may be calculated as the position of the work device 330. Then, the display control unit 222 plots the tip position of the attachment 333 in the three-dimensional local coordinate space Q (FIG. 2) of the construction machine 30 centered on, for example, the position of the center of gravity of the construction machine 30 to plot the tip position of the attachment 333. The position may be calculated.

The operation mechanism 15 is composed of an actuator, and a control signal for generating an operation force according to the operation amount received by the communication unit 11 is input to generate the operation force according to the operation amount, and the construction machine 30 It is applied to the operation lever 31. For example, assuming that the operating lever 31 is composed of an ATT lever, a traveling lever, and a swivel lever, the operating mechanism 15 is composed of three operating mechanisms corresponding to each operating lever. Further, in this example, when the ATT lever is composed of an operation lever that can be tilted in four directions of front / rear / left / right, the operation mechanism 15 has an actuator that tilts the ATT lever in the front / rear direction and an actuator that tilts the ATT lever in the left / right direction. It is equipped with an actuator to make it.

The camera 14 is composed of, for example, a CMOS sensor or a CCD sensor, and acquires a peripheral image showing the surroundings of the construction machine 30 by periodically taking a picture of the surroundings of the construction machine 30 from inside the cockpit 3C at a predetermined sampling cycle. do. Here, the camera 14 is attached to the inside of the driver's seat 3C so that the line of sight faces the front of the driver's seat 3C, and an image of the surroundings of the construction machine 30 is displayed through the front glass that closes the front of the driver's seat 3C. Take a picture. That is, if the operator is seated in the driver's seat 3C, the camera 14 captures the scenery that the operator sees through the front glass.

When the communication unit 11 receives the operation amount of the operation lever 24, the controller 12 outputs a control signal corresponding to the operation amount to the operation mechanism 15. As a result, the slave 10 acts as a dummy of the operator who operates the master 20 and directly operates the construction machine 30.

The construction machine 30 includes an operation lever 31 (an example of an operation member) and a controller 32. The operation lever 31 is provided in the cockpit 3C of the construction machine 30, and is tilted by applying an operation force from the operation mechanism 15 of the slave 10. Like the operating lever 24 of the master 20, the operating lever 31 is composed of at least one or a plurality of an ATT lever, a traveling lever, and a swivel lever. In this case, at least one or more of the ATT lever and the swivel lever constituting the operation lever 31 are operation levers that can be tilted in four directions of front, back, left and right, like the ATT lever and the swivel lever that constitute the operation lever 24 of the master 20. It is composed. Further, the traveling lever constituting the operating lever 31 is composed of an operating lever that can be tilted in two directions in the front-rear direction, like the traveling lever constituting the operating lever 24 of the master 20.

The controller 32 activates the decelerator function that idles the engine of the construction machine 30 when the operating lever 31 has not been operated for a certain period of time.

FIG. 3 is a diagram showing an example of a display screen G3 displayed on the display device 23 of the master 20. The display screen G3 includes a peripheral image G31 and a margin area G32 that covers the periphery of the peripheral image G31. In the example of FIG. 3, the outer frame shape of the peripheral image G31 is formed by simulating the shape of the front glass of the driver's seat 3C and having a rectangular shape whose width gradually becomes shorter toward the lower side.

The table shown on the upper side of FIG. 3 is a table summarizing the types of display objects used in the present embodiment. In the present embodiment, any one of the two-dimensional arrow image OA, the three-dimensional arrow image OB, the construction machine image OC, and the work equipment image OD is adopted as the display object.

The two-dimensional arrow image OA is an arrow-shaped image that two-dimensionally shows the tip position of the attachment 333 of the work device 330 with respect to the cockpit 3C. With reference to FIG. 2, in the local coordinate space Q of the work apparatus 330, the origin is set at, for example, the center of gravity of the construction machine 30, the Y axis is set in the height direction, the X axis is set in the left-right direction, and the Z axis is set in the front-rear direction. ing. The X-axis, Y-axis, and Z-axis are orthogonal to each other.

In FIG. 2, when the tip position of the attachment 333 is located at the point P1, the display control unit 222 calculates the three-dimensional vector V1 from the predetermined reference position O1 in the local coordinate space Q toward the point P1. Then, the display control unit 222 obtains a projection vector of the three-dimensional vector V1 on the XY plane parallel to the front surface of the cockpit 3C, and displays an arrow-shaped image having a predetermined length pointing in the same direction as the projection vector as a two-dimensional arrow. Generated as image OA. Then, the display control unit 222 displays the two-dimensional arrow image OA at a predetermined display position on the display screen G3. As the reference position O1, a position on the local coordinate space Q corresponding to the display position of the two-dimensional arrow image OA on the display screen G3 is adopted. In FIG. 3, a predetermined position on the upper left of the surrounding image G31 is a display position of the two-dimensional arrow image OA. Therefore, in this case, the X and Y components of the reference position O1 are, for example, the X and Y coordinates of the local coordinate space Q corresponding to this display position, and the Z component of the reference position O1 is, for example, the front glass. Coordinates are adopted.

The three-dimensional arrow image OB is an arrow-shaped image that three-dimensionally shows the tip position of the attachment 233 of the work device 330 with respect to the cockpit 3C. Here, the three-dimensional arrow image OB is formed into a cuneus-shaped cuneus OB1 and a cuneus OB1 in which each of the two triangular prisms extending in the diagonal direction is cut at a sharp angle with respect to the longitudinal direction and the cut surfaces are bonded to each other. It is an image obtained by projecting a three-dimensional model including a sandwiched spherical portion OB2 on a virtual screen, and the direction of the working device 330 is indicated by the direction of the tip E1 of the cuneus portion OB1. The virtual screen is set at a position in the local coordinate space Q corresponding to the front glass of the driver's seat 3C, for example.

The display control unit 222 obtains the 3D vector V1 as in the case of the 2D arrow image OA, so that the tip E1 of the 3D model of the 3D arrow image OB faces the end point (point P1) of the 3D vector V1. The three-dimensional model is placed at the reference position O1. Then, the display control unit 222 generates a three-dimensional arrow image OB by projecting the arranged three-dimensional model on the virtual screen, and displays it on the display screen G3. In the example of FIG. 3, since the center position of the surrounding image G31 is the display position of the three-dimensional arrow image OB, the X component and the Y component of the reference position O1 correspond to, for example, the X component and the Y component of the center position. The X and Y coordinates of the local coordinate space Q are adopted, and the Z component is, for example, the Z coordinate of the local coordinate space Q separated forward by a predetermined distance from the front glass.

When the display control unit 222 adopts the two-dimensional arrow image OA or the three-dimensional arrow image OB, the two-dimensional arrow image OA or the three-dimensional arrow image as the tip position of the attachment 333 of the work device 330 approaches the cockpit 3C. The size of the OB may be displayed larger. In this case, the display control unit 222 prepares in advance a table or function that defines the relationship between the two so that the smaller the norm of the three-dimensional vector V1 is, the larger the enlargement ratio is, and three-dimensionally using this table or function. The magnification corresponding to the magnitude of the vector V1 is determined. Then, the display control unit 222 multiplies the three-dimensional model of the two-dimensional arrow image OA or the three-dimensional arrow image OB having the default size by the determined enlargement ratio, so that the two-dimensional arrow image OA or the three-dimensional arrow The three-dimensional model of the image OB may be enlarged. The enlargement ratio has a value of 0 or more, and when it is 1 or less, the two-dimensional arrow image OA or the three-dimensional arrow image OB is reduced.

See FIG. The construction machine image OC is an image simulating the actual posture of the construction machine 30 from the side view. The display control unit 222 uses the boom angle, arm angle, attachment angle, and turning angle included in the detection values of the angle sensor 13 to correct the image of the construction machine 30 in the default posture from the side view, thereby constructing the construction machine. Generate a machine image OC. Then, the display control unit 222 displays the generated construction machine image OC at a predetermined display position on the display screen G3. In the example of FIG. 3, the construction machine image OC is displayed at the left center position in the margin area G32. As the default posture, for example, a posture in which the lower traveling body 310 and the upper swivel body 320 are both facing the front and the boom angle, the arm angle, and the attachment angle are each at a predetermined default angle can be adopted.

The work device image OD is an image showing the current posture of the work device 330 with respect to the upper swing body 320. The display control unit 222 is provided with a work device image table in which the work device image OD according to the posture of the work device 330 is registered in association with the boom angle, the arm angle, and the attachment angle. Then, the display control unit 222 reads the work device image OD corresponding to the combination of the boom angle, the arm angle, and the attachment angle included in the detection value of the angle sensor 13 from the work device image table, and determines the predetermined value on the display screen G3. It may be displayed at the position of. In the example of FIG. 3, the working device image OD is displayed in the center of the upper part of the surrounding image G31.

FIG. 4 is a diagram showing a pattern of display positions of display objects. The table shown on the upper side of FIG. 4 is a table summarizing the patterns of display positions. On the display screen G3 shown in FIG. 4, three patterns PA, PB, and PC are shown. The pattern PA is a pattern in which the entire area of the display object OBX is superimposed and displayed on the surrounding image G31. The pattern PB is a pattern in which a part of the display object OBX is superimposed and displayed on the surrounding image G31, and the rest of the display object OBX is superimposed and displayed on the margin area G32. The pattern PC is a pattern in which the entire area of the display object OBX is superimposed and displayed on the margin area G32.

The display control unit 222 displays the display object OBX in any one of the patterns PA to PC. In the example of FIG. 4, as the pattern PA, a pattern for displaying the display object OBX composed of the three-dimensional arrow image OB in the center of the surrounding image G31 is shown. Further, as the pattern PB, a pattern is shown in which a display object OBX composed of a three-dimensional arrow image OB is partially superimposed and displayed on the surrounding image G31 on the upper side of the surrounding image G31. Further, as the first pattern PC, a pattern is shown in which the entire area of the display object OBX composed of the three-dimensional arrow image OB is displayed on the upper side of the margin area G32. Further, as the second pattern PC, a pattern is shown in which the entire area of the display object OBX composed of the construction machine image OC is displayed on the left side of the margin area G32.

In the example of FIG. 4, the three-dimensional arrow image OB is used as the pattern PA, PB, and the first pattern PC, but this is an example, and the two-dimensional arrow image OA shown in FIG. 3 is constructed. The machine image OC or the working device image OD may be adopted. Further, although the construction machine image OC is used as the second pattern PC, the two-dimensional arrow image OA, the three-dimensional arrow image OB, or the working device image OD may be adopted. When the two-dimensional arrow image OA and the three-dimensional arrow image OB are adopted, the coordinates of the reference position O1 shown in FIG. 2 are the coordinates of the local coordinate space Q corresponding to the display positions shown by the patterns PA to PC. NS.

Further, the display position of the display object OBX in the pattern PA is not limited to the center of the peripheral image G31 shown in FIG. 4, but is within the peripheral image G31 and at a position where the display object OBX does not protrude from the peripheral image G31. If so, any position may be adopted. Further, the display position of the display object OBX in the pattern PB is not limited to the position straddling the upper edge of the peripheral image G31 shown in FIG. 4, and the position straddling the left edge, the right edge, or the lower edge of the peripheral image G31 is adopted. May be done. Further, the display position of the display object OBX on the pattern PC is not limited to the upper margin area G32 shown in FIG. 4, and the left side, right side, or lower margin area may be adopted.

FIG. 5 is a table summarizing the display / non-display timing of the display object OBX according to the present embodiment. In the present embodiment, there are seven display / non-display timing patterns of the display object OBX. In FIG. 5, the “display system” indicates whether or not the display object OBX is displayed on the display device 23. "Operation of the operation lever" indicates whether or not the operation lever 24 is operated by the operator. The “position of the working device” indicates whether or not the working device 330 is included in the surrounding image G31. The “non-operation elapsed time” indicates whether or not the elapsed time since the operation lever 24 is no longer operated has elapsed for a certain period of time or more. The "system state" indicates whether or not a communication connection has been established between the slave 10 and the master 20. Further, in the "system state", "communication start" indicates the state immediately after the communication between the slave 10 and the master 20 is started. "Decelerator function" indicates whether or not the decelerator function is in operation. The “blocking lever” indicates whether or not the blocking lever 25 is locked.

In the pattern "1", when the operation lever 24 is operated, the display control unit 222 does not display the display object OBX regardless of whether or not the work device 330 is included in the surrounding image G31. This is because even if the work device 330 disappears from the surrounding image G31 during the operation of the operation lever 24, the operator can predict the position of the work device 330 to some extent from the current operation, and displaying the display object OBX rather hinders the work. It is based on the idea that it will be.

In the pattern "2", the display control unit 222 does not display the display object OBX when the work device 330 is included in the surrounding image G31. This is based on the idea that when the surrounding image G31 includes the work device 330, the operator can recognize the position of the work device 330 without displaying the display object OBX.

In the pattern "3", when the surrounding image G31 does not include the work device 330, the display control unit 222 does not display the display object OBX until the operation lever 24 is not operated for a certain period of time or longer. Continue the display.

The pattern "4" is the inside out of the pattern "3", and the display control unit 222 continues the state in which the operation lever 31 is not operated for a certain period of time or more when the work device 330 is not included in the surrounding image G31. After that, the display object OBX is displayed. This is thought to prevent the occurrence of chattering in which the display and non-display of the display object OBX are repeated in a short time in a scene where the appearance and disappearance of the work device 330 are repeated in a short time with respect to the surrounding image G31. It is based on. As the fixed time, appropriately set values such as 5 seconds, 10 seconds, 30 seconds, and 1 minute can be adopted.

In the pattern "5", when the surrounding image G31 does not include the work device 330 at the start of communication between the master 20 and the slave 10, the display control unit 222 regardless of whether or not the operation lever 24 is operated. , Display the display object OBX. This is based on the idea that if the work device 330 does not appear in the surrounding image G31 at the start of communication, the operator cannot recognize the position of the work device 330.

In the pattern "6", when the shutoff lever 25 is locked and the surrounding image G31 does not include the work device 330, the display control unit 222 regardless of whether or not the operation lever 24 is operated. , Display the display object OBX.

This is because when the shutoff lever 25 is locked and the work of the work device 330 is suspended, if the work device 330 is not included in the surrounding image G31, the display object OBX is displayed and the operator is displayed. It is based on the idea that it is preferable to make the work device 330 recognize the position of the work device 330.

In the pattern "7", when the decelerator function is operating and the surrounding image G31 does not include the work device 330, the display control unit 222 displays the display object regardless of whether or not the operation lever 24 is operated. Display OBX. This is because when the decelerator function is operating and the work by the work device 330 is suspended, if the work device 330 is not included in the surrounding image G31, the display object OBX is displayed and the operator is made to display the work device 330. It is based on the idea that it is desirable to recognize the position of.

FIG. 6 is a flowchart showing an example of processing of the remote control system according to the embodiment of the present invention. The flowchart of FIG. 6 is repeatedly executed. In S101, the determination unit 221 determines whether or not the work device 330 of the surrounding image G31 is included. When the work device 330 is included in the surrounding image G31 (YES in S101), the display control unit 222 does not display the display object OBX on the display screen G3 (S107).

When the work device 330 is not included in the surrounding image G31 (NO in S101), the display control unit 222 determines whether or not the shutoff lever 25 is in the locked position from the detection result of the sensor S2 (S102). When it is determined that the shutoff lever 25 is in the locked position (YES in S102), the display control unit 222 displays the display object OBX on the display screen G3 (S103). As a result, the pattern "6" is realized.

On the other hand, when it is determined that the shutoff lever 25 is not in the locked position (NO in S102), the display control unit 222 determines whether or not the decelerator function is operating (S104). When it is determined that the decelerator function is operating (YES in S104), the display control unit 222 displays the display object OBX on the display screen G3 (S103). As a result, the pattern "7" is realized.

On the other hand, when it is determined that the decelerator function is not operating (NO in S104), the display control unit 222 determines whether or not the operating lever 24 is operating from the detection result of the sensor S1 (S105). Here, the display control unit 222 determines that the operation is in progress if the operation amount of the operation lever 24 detected by the sensor S1 is not 0, and if the operation amount of the operation lever 24 detected by the sensor S1 is 0. , It may be determined that the operation is not in progress.

When the operation lever 24 is being operated (YES in S105), the display control unit 222 does not display the display object on the display screen G3 (S107). On the other hand, when the operation lever 24 is not being operated (NO in S105), the display control unit 222 determines whether or not a certain time or more has elapsed since the operation of the operation lever 24 was stopped (S106). When a certain time or more has passed since the operation of the operation lever 24 was stopped (YES in S106), the display control unit 222 displays the display object OBX on the display screen G3 (S103). On the other hand, if a certain time or more has not passed since the operation of the operation lever 24 was stopped (NO in S106), the display control unit 222 does not display the display object OBX on the display screen G3 (S107). When NO in S105, YES in S106, and the work device 330 disappears from the surrounding image G31 due to the flow of S103, the display object OBX is displayed on the display screen after a certain period of time has passed while the operation lever 24 is not being operated. It will be displayed on G3. As a result, the patterns "3" and "4" are realized.

Further, according to the flow of YES and S107 in S101 and the flow of YES and S107 in S105, the display object OBX is displayed during the operation of the operation lever 24 regardless of whether or not the work device 330 is present in the surrounding image G31. Will not be done. As a result, the pattern "1" is realized.

If YES in S101, the process proceeds to S107. As a result, the pattern "2" is realized.

FIG. 7 is a flowchart showing an example of processing at the start of communication of the remote control system according to the embodiment of the present invention. In S201, the display control unit 222 of the master 20 determines whether or not the communication between the master 20 and the slave 10 has been started. Here, for the start of communication, for example, after the operator of the master 20 turns on the power of the master 20, a connection request transmission instruction is input to the master 20 input device (not shown), and the master 20 sends the connection request to the slave. It is done by transmitting to 10. Here, the connection request includes the identification information of the slave 10. The slave 10 that has received the connection request performs an authentication process based on the identification information, and if the authentication is permitted, sends a connection response to the master 20. As a result, the communication connection between the master 20 and the slave 10 is established, and communication between the two is started.

When communication is started (YES in S201), the process proceeds to S202, and if communication is not started (NO in S201), the process waits in S201.

In S202, the determination unit 221 determines whether or not the work device 330 is included in the surrounding image G31. When the work device 330 is included in the surrounding image G31 (YES in S202), the display control unit 222 does not display the display object OBX on the display screen G3 (S204). On the other hand, when the work device 330 is not included in the surrounding image G31 (NO in S202), the display control unit 222 displays the display object OBX on the display screen G3 (S203). As a result, when the work device 330 is not included in the surrounding image G31 at the start of communication, the display object OBX is displayed and the pattern "5" is realized.

As described above, according to the present embodiment, when the work device 330 is not included in the peripheral image G31 of the construction machine 30 taken from the cockpit 3C of the construction machine 30, the position of the work device 330 is set to the operator. The display object OBX for notifying is displayed on the display screen of the master 20.

Therefore, even if the work device 330 disappears from the surrounding image G31 displayed on the display device 23 of the master 20, the operator of the master 20 can recognize the position of the work device 330. As a result, the operator can operate the work device 330 with peace of mind, and the work load of the operator can be reduced.

Further, since the flow of FIG. 6 is repeatedly executed, the display object OBX is displayed in real time on the display screen G3.

In order to prevent the work device 330 from disappearing from the surrounding image G31, it is conceivable to install an omnidirectional camera in the cockpit 3C and take an omnidirectional surrounding image. However, in this method, since the amount of data of the surrounding image increases, the communication traffic between the main operating device and the suboperating device increases, and there arises a problem that the screen of the master display device becomes large.

In the present embodiment, even if the omnidirectional camera is not installed, when the working device 330 disappears from the surrounding image G31, the position of the working device 330 can be notified to the operator, so that such a problem can be solved.

The following modifications can be adopted in the present invention.

(1) In FIG. 4, when a pattern PC for displaying the display object OBX in the margin area G32 is adopted, the display control unit 222 displays the display object OBX regardless of whether or not the surrounding image G31 includes the work device 330. May be displayed at all times. In particular, when the construction machine image OC or the work device image OD is adopted as the display object OBX, it is preferable to always display the display object OBX because the operator can have a bird's-eye view of the posture of the construction machine 30.

(2) In FIG. 4, a pattern PC that constantly displays a construction machine image OC or a work device image OD as a display object OBX in a margin area G32, and a surrounding image G31 for the entire area of the two-dimensional arrow image OA or the three-dimensional arrow image OB. A display mode in which the pattern PA to be superimposed and displayed is combined may be adopted. In this case, the two-dimensional arrow image OA or the three-dimensional arrow image OB may be displayed or hidden according to the patterns "1" to "7" shown in FIG.

3C Driver's seat 10 Slave 11 Communication unit 12 Controller 13 Angle sensor 14 Camera 15 Operation mechanism 20 Master 21 Communication unit 22 Controller 23 Display device 24 Operation lever 25 Blocking lever 30 Construction machine 31 Operation lever 32 Controller 100 Communication path 221 Judgment unit 222 Display Control unit 233 Attachment G3 Display screen G31 Surrounding image G32 Margin area OBX Display object OA 2D arrow image OB 3D arrow image OC Construction machine image OD Work equipment image

Claims (16)

A remote control system that remotely controls construction machinery, including work equipment.
It is provided with a main operating device for remotely controlling the construction machine and a secondary operating device mounted on the construction machine and operating the working device based on an operation received by the main operating device.
The slave operation device is
A camera that acquires a surrounding image of the surroundings of the construction machine from inside the cockpit of the construction machine, and
A posture detection unit that detects the posture of the work device, and
It is provided with a first communication unit that transmits the surrounding image and the posture information indicating the posture of the work device detected by the posture detection unit to the main operation device.
The main operating device is
An operating member that accepts operator operations and
A second communication unit that receives the surrounding image and the posture information transmitted from the slave operation device, and
A display device that displays a display screen including the surrounding image received by the second communication unit, and
A determination unit that determines whether or not the work device is included in the surrounding image received by the second communication unit, and a determination unit.
When the determination unit determines that the surrounding image does not include the work device, the position of the work device is calculated from the posture information received by the second communication unit, and the calculated position is used as the operator. A remote control system including a display control unit that displays a display object for notifying the display on the display screen. The remote control system according to claim 1, wherein the display control unit superimposes and displays at least a part of the display object on the surrounding image. The remote control system according to claim 2, wherein the display control unit stops displaying the display object when the determination unit determines that the surrounding image includes the work device. The remote control system according to claim 2 or 3, wherein the display control unit stops displaying the display object when the operation member is operated, regardless of the determination result by the determination unit. When the determination unit determines that the surrounding image does not include the work device, the display control unit determines that the operation member is not operated for a certain period of time or longer, and then the display object is displayed. The remote control system according to any one of claims 2 to 4. When the determination unit determines that the surrounding image does not include the work device at the start of communication between the main operation device and the slave operation device, the display control unit operates the operation member. The remote control system according to any one of claims 2 to 5, which displays the display object regardless of whether or not the display object is displayed. The main operating device further includes a shutoff lever that receives an operation from an operator for shutting off the operation of the operating member.
When the display control unit determines that the shutoff lever is locked and the determination unit does not include the work device in the surrounding image, the display control unit determines whether or not the operation member is operated. The remote control system according to any one of claims 2 to 6, which displays the display object regardless of the display object. The construction machine has a decelerator function that idles the engine of the construction machine when the operating member has not been operated for a certain period of time.
Any of claims 2 to 7, wherein the display control unit displays the display object when the decelerator function is operating and the determination unit determines that the work device is not included in the surrounding image. Remote control system described in Crab. The remote control system according to any one of claims 1 to 8, wherein the display object is composed of an arrow image that three-dimensionally indicates the direction in which the work device is located. The remote control system according to any one of claims 1 to 8, wherein the display object is composed of an arrow image that two-dimensionally indicates the direction in which the work device is located. The remote control system according to any one of claims 1 to 8, wherein the display object is composed of a construction machine image simulating the actual posture of the construction machine from a side view. The remote control system according to any one of claims 1 to 8, wherein the display object is composed of a work device image simulating the actual posture of the work device. The remote control system according to claim 9 or 10, wherein the display control unit displays the size of the arrow image larger as the position of the work device approaches the cockpit. The remote control system according to claim 9 or 10, wherein the display control unit directs the tip of the arrow image in a direction in which the work device is located with respect to the cockpit in a direction projected on the surrounding image. The remote control system according to claim 1, wherein the display control unit displays the peripheral image on the display device so as to include a margin area adjacent to the peripheral image, and displays the display object in the margin area. .. It is the main operating device in a remote control system for remotely controlling construction machinery including work equipment.
An operating member that accepts operator operations and
A communication unit that receives a surrounding image of the surroundings of the construction machine and posture information indicating the posture of the work device transmitted from the cockpit of the construction machine transmitted from the slave operation device communicably connected to the main operation device. When,
A display device that displays a display screen including the surrounding image received by the communication unit, and
A determination unit that determines whether or not the image of the work device is included in the surrounding image received by the communication unit, and a determination unit.
When the determination unit determines that the surrounding image does not include the work device, the position of the work device in the construction machine is calculated from the posture information received by the communication unit, and the calculated position is calculated. A main operating device including a display control unit that displays a display object for notifying the operator on the display screen.

Images