JP7310637B2 - REMOTE OPERATION SUPPORT SERVER, REMOTE OPERATION SUPPORT SYSTEM AND REMOTE OPERATION SUPPORT METHOD - Google Patents

Description

TECHNICAL FIELD The present invention relates to technology for supporting remote control of a work machine using a remote control device.

When a two-dimensional image of the work site is displayed on the display device, it is difficult for the worker to perceive the perspective of the work site. As a result, the operator cannot smoothly perform the remote operation, and the work efficiency of the work machine may decrease. 2. Description of the Related Art A display system has been proposed that enables a worker to effectively perceive a perspective of a work site in remote operation of a work machine (see, for example, Patent Document 1). According to this display system, a free viewpoint image of an object is displayed based on image data including three-dimensional data of the object at the work site and the viewpoint position data of the worker.

JP 2018-152738 A

However, only by displaying an image according to an operator's arbitrary viewpoint, for example, the operator cannot recognize the relative positions of the end effector such as the bucket of the excavator and the dump truck bed in the vicinity. Therefore, the present invention provides a technique that allows an operator remotely operating a working machine to intuitively recognize the relative positional relationship between the end effector of the working machine and objects around it. intended to provide

A remote operation support server of the present invention is a remote control device for a working machine having a lower traveling body, an upper revolving body capable of turning with respect to the lower traveling body, and a work attachment capable of moving an end effector with respect to the upper revolving body. a remote operation support server for supporting remote operation using a first support process for recognizing an index value representing the distance between the upper rotating body and the end effector based on communication with the work machine a second support processing element that causes an output interface constituting the remote control device to output an indicator representing the index value recognized by the first support processing element based on communication between the element and the remote control device; characterized by comprising
Preferably, the first support processing element recognizes the index value representing the swing angle of the upper swing body relative to the undercarriage based on communication with the work machine.

According to the remote operation support server having this configuration, the length of the distance between the end effector of the working machine and the upper swing body , or the length of the distance and the lower travel distance can be detected through the indicator output to the output interface that constitutes the remote operation device. The operator of the remote control device can intuitively recognize the turning angle of the upper turning body with respect to the body. In addition, the operator of the remote control device can intuitively know the length of the distance between the end effector of the work machine and the upper revolving body and/or the time change of the turning angle of the upper revolving body with respect to the lower traveling body through the time change of the indicator. can be recognized.

In the remote operation support server of the present invention, the first support processing element calculates the index value at least one of the start time and end time of the interaction between the end effector and the object based on communication with the work machine. It is preferable to recognize it as a reference index value.

According to the remote operation support server having this configuration, the interaction between the end effector and the object is started by comparing the indicator representing the current index value with the indicator representing the reference index value output to the output interface as a reference. Based on at least one of the time point and the end time point, the length of the distance between the current end effector and the upper swing body and/or the deviation of the swing angle of the upper swing body with respect to the lower traveling body is reported to the operator of the remote control device. It can be recognized intuitively.

In the remote operation support server of the present invention, it is preferable that the second support processing element causes the output interface to output a reference gauge representing at least one of an upper limit value and a lower limit value of the index value together with the indicator.

According to the remote operation support server with this configuration, by comparing the indicator and the reference gauge output to the output interface, it is possible to determine how close the index value represented by the indicator is to its upper limit and/or lower limit. can be intuitively recognized by the operator of the remote control device.

1 is an explanatory diagram regarding the configuration of a remote operation support system as an embodiment of the present invention; FIG. Explanatory drawing about the structure of a remote control. Explanatory drawing about the structure of a working machine. Explanatory drawing about the 1st function of a remote operation assistance system. Explanatory drawing about the 2nd function of a remote operation assistance system. FIG. 10 is an explanatory diagram regarding the interval between the end effector of the working machine and the upper revolving body; Explanatory drawing about the 1st indicator showing a 1st index value. FIG. 4 is an explanatory diagram regarding the turning angle of the upper revolving body with respect to the lower traveling body of the work machine; Explanatory drawing about the 2nd indicator showing a 2nd index value. Explanatory drawing about the 1st operating environment of a working machine. Explanatory drawing regarding the 1st display mode of an environment image and an indicator. Explanatory drawing about the 2nd operating environment of a working machine. Explanatory drawing about the 2nd display mode of an environment image and an indicator. Explanatory drawing about the 3rd operating environment of a working machine. Explanatory drawing regarding the 3rd display mode of an environment image and an indicator. Explanatory drawing about the 4th operating environment of a working machine. Explanatory drawing regarding the 4th display mode of an environment image and an indicator. FIG. 10 is an explanatory diagram relating to a fifth operating environment of the work machine; Explanatory drawing about the 5th display mode of an environment image and an indicator. FIG. 11 is an explanatory diagram regarding a sixth operating environment of the working machine; Explanatory drawing about the sixth display mode of the environment image and the indicator.

(Configuration of remote operation support system)
A remote operation support system as one embodiment of the present invention shown in FIG. there is Remote operation support server 10, remote operation device 20, and work machine 40 are configured to be able to communicate with each other through a network. The mutual communication network between remote operation support server 10 and remote operation device 20 and the mutual communication network between remote operation support server 10 and work machine 40 may be the same or different.

(Configuration of remote operation support server)
The remote operation support server 10 includes a database 102 , a first support processing element 121 and a second support processing element 122 . The database 102 stores and holds captured image data and the like. The database 102 may be configured by a database server separate from the remote operation support server 10 . Each support processing element is composed of an arithmetic processing unit (single-core processor or multi-core processor or a processor core that constitutes it), reads necessary data and software from a storage device such as memory, and processes the data into the software. Accordingly, arithmetic processing, which will be described later, is executed. The remote operation support server 10 may be configured with a remote operation device 20 . In this case, remote control device 200 may comprise first support processing element 121 and second support processing element 122 .

(Configuration of remote control device)
The remote control device 20 includes a remote control device 200 , a remote input interface 210 and a remote output interface 220 . The remote control device 200 is configured by an arithmetic processing device (single-core processor or multi-core processor or a processor core constituting this), reads necessary data and software from a storage device such as a memory, and uses the data as a target for the software. Execute the arithmetic processing accordingly. The remote input interface 210 has a remote control mechanism 211 . The remote output interface 220 comprises an image output device 221 and a remote wireless communication device 222 .

The remote control mechanism 211 includes a traveling operating device, a turning operating device, a boom operating device, an arm operating device, and a bucket operating device. Each operating device has an operating lever that receives a rotating operation. An operating lever (running lever) of the operating device for running is operated to move the lower running body 410 of the work machine 40 . The travel lever may also serve as a travel pedal. For example, a traction pedal may be provided that is fixed to the base or lower end of the traction lever. An operation lever (swing lever) of the swing operation device is operated to move a hydraulic swing motor that constitutes the swing mechanism 430 of the work machine 40 . An operating lever (boom lever) of the boom operating device is operated to move the boom cylinder 442 of the work machine 40 . An operating lever (arm lever) of the arm operating device is operated to move the arm cylinder 444 of the working machine 40 . An operating lever (bucket lever) of the bucket operating device is operated to move the end effector cylinder 446 of the work machine 40 .

Each control lever that constitutes the remote control mechanism 211 is arranged around a seat St on which the operator sits, as shown in FIG. 2, for example. The seat St is in the form of a high-back chair with armrests, a low-back chair without a headrest, or a chair without a backrest. may be

A pair of left and right travel levers 2110 corresponding to the left and right crawlers are arranged in left and right in front of the seat St. One control lever may serve as a plurality of control levers. For example, the left operation lever 2111 provided in front of the left frame of the seat St shown in FIG. 2 functions as an arm lever when operated in the longitudinal direction, and when operated in the lateral direction. function as a pivot lever. Similarly, the right operation lever 2112 provided in front of the right frame of the seat St shown in FIG. It may function as a bucket lever in some cases. The lever pattern may be arbitrarily changed by an operator's operation instruction.

For example, as shown in FIG. 2, the image output device 221 includes a central image output device 2210 having substantially rectangular screens arranged in front of the sheet St, diagonally forward left and diagonally forward right, and a left image output device 2210 . It is composed of an output device 2211 and a right image output device 2212 . The shapes and sizes of the respective screens (image display areas) of the central image output device 2210, the left image output device 2211 and the right image output device 2212 may be the same or different.

As shown in FIG. 2, the left image output device 2211 is tilted such that the screen of the central image output device 2210 and the screen of the left image output device 2211 form an inclination angle θ1 (for example, 120°≦θ1≦150°). is adjacent to the left edge of central image output device 2210 . As shown in FIG. 2, the right image output device 2212 is tilted such that the screen of the central image output device 2210 and the screen of the right image output device 2212 form an inclination angle θ2 (for example, 120°≦θ2≦150°). is adjacent to the right edge of central image output device 2210 . The tilt angles θ1 and θ2 may be the same or different.

The respective screens of the central image output device 2210, the left image output device 2211, and the right image output device 2212 may be parallel to the vertical direction or may be inclined with respect to the vertical direction. At least one of the central image output device 2210, the left image output device 2211, and the right image output device 2212 may be composed of a plurality of divided image output devices. For example, the central image output device 2210 may be composed of a pair of vertically adjacent image output devices each having a substantially rectangular screen. The image output devices 2210 to 2212 may further include speakers (audio output devices).

(Configuration of working machine)
The work machine 40 includes a real machine control device 400 , a real machine input interface 41 , a real machine output interface 42 , and an operating mechanism 440 . The actual device control device 400 is composed of an arithmetic processing device (single-core processor or multi-core processor or a processor core that constitutes it), reads necessary data and software from a storage device such as a memory, and processes the data into the software. Execute the arithmetic processing accordingly.

The work machine 40 is, for example, a crawler-type liftmag (construction machine), and as shown in FIG. and an upper rotating body 420 mounted thereon. A cab 424 (driver's cab) is provided on the front left side of the upper swing body 420 . A work attachment 440 is provided at the front central portion of the upper swing body 420 .

The real machine input interface 41 includes a real machine operating mechanism 411 , a real machine imaging device 412 , and a state sensor 414 . The actual machine operating mechanism 411 includes a plurality of operating levers arranged around a seat arranged inside the cab 424 in the same manner as the remote operating mechanism 211 . The cab 424 is provided with a drive mechanism or a robot that receives a signal corresponding to the operation mode of the remote control lever and moves the actual machine control lever based on the received signal. The actual machine imaging device 412 is installed inside the cab 424, for example, and images the environment including at least part of the operating mechanism 440 through the front window and the pair of left and right side windows. Some or all of the front window and side windows may be omitted. The state sensor 414 detects the rotation angle of the boom 441 with respect to the upper swing body 420, the rotation angle of the arm 443 with respect to the tip of the boom 441, the rotation angle of the end effector 445 with respect to the tip of the arm 443, and the rotation angle of the swing mechanism 430. It is composed of an angle sensor (for example, a rotary encoder) for measuring each turning angle.

The real machine output interface 42 includes a real machine wireless communication device 422 .

A work attachment 440 as an operating mechanism includes a boom 441 attached to the upper rotating body 420 so as to be able to rise and fall, an arm 443 rotatably connected to the tip of the boom 441, and a tip of the arm 443 rotatable. and an end effector 445 comprising an electromagnet or magnet coupled to the end effector 445 . The work attachment 440 is equipped with a boom cylinder 442, an arm cylinder 444, and an end effector cylinder 446, which are configured by telescopic hydraulic cylinders. The end effector 445 may consist of a grapple or nibbler.

The boom cylinder 442 is interposed between the boom 441 and the upper slewing body 420 so as to expand and contract when supplied with hydraulic oil to rotate the boom 441 in the hoisting direction. The arm cylinder 444 is interposed between the arm 443 and the boom 441 so as to expand and contract when supplied with hydraulic oil to rotate the arm 443 about the horizontal axis with respect to the boom 441 . The end effector cylinder 446 is interposed between the end effector 445 and the arm 443 so as to expand and contract when supplied with hydraulic oil to rotate the end effector 445 with respect to the arm 443 about the horizontal axis.

(first function)
The first function of the remote operation support system configured as described above will be described with reference to the flow chart shown in FIG. In the flowchart, the block "C●" is used for simplification of the description, means transmission and/or reception of data, and processing in the branch direction is executed on the condition of transmission and/or reception of the data. It means a conditional branch.

In the remote operation device 20, it is determined whether or not the operator has performed a designated operation through the remote input interface 210 (FIG. 4/STEP 210). The “specifying operation” is, for example, an operation such as tapping on the remote input interface 210 for specifying the work machine 40 intended for remote operation by the operator. If the determination result is negative ( FIG. 4 /STEP 210 . . . NO), the processing after determining whether or not there is a designation operation is repeated. On the other hand, if the determination result is affirmative (FIG. 4/STEP 210 . . . YES), an environment confirmation request is transmitted to the remote operation support server 10 through the remote wireless communication device 222 (FIG. 4/STEP 212).

When the remote operation support server 10 receives the environment confirmation request, the first support processing element 121 transmits the environment confirmation request to the corresponding work machine 40 (FIG. 4/C110).

In the work machine 40, when an environment confirmation request is received through the real machine wireless communication device 422 (Fig. 4/C410), the real machine control device 400 acquires a captured image through the real machine imaging device 412 (Fig. 4/STEP 410). Captured image data representing the captured image is transmitted to remote control device 20 through actual wireless communication device 422 by actual device control device 400 (FIG. 4/STEP 411).

In the remote operation support server 10, when the captured image data is received by the first support processing element 121 (FIG. 4/C111), the environment image data corresponding to the captured image is sent to the remote control device 20 by the second support processing element 122. (FIG. 4/STEP 110). The environmental image data is image data representing a simulated environmental image generated based on the captured image, in addition to the captured image data itself.

When the remote control device 20 receives environmental image data through the remote wireless communication device 222 (FIG. 4/C210), the remote control device 200 outputs an environmental image corresponding to the environmental image data to the image output device 221. (FIG. 4/STEP 224).

As a result, for example, as shown in FIG. 8B, an environmental image including an arm 443 and an end effector 445 that are part of the work attachment 440, and the dump truck 50 and its loading platform 52 is displayed by the image output device 221. output to

In the remote operation device 20, the operation mode of the remote control mechanism 211 is recognized by the remote control device 200 (FIG. 4/STEP 216), and a remote operation command corresponding to the operation mode is sent through the remote wireless communication device 222 as remote operation support. It is transmitted to the server 10 (FIG. 4/STEP 218).

In the remote operation support server 10, when the remote operation command is received by the second support processing element 122, the remote operation command is transmitted to the work machine 40 by the first support processing element 121 (FIG. 4/ C112).

In the work machine 40, when the actual machine controller 400 receives an operation command through the actual machine wireless communication device 422 (FIG. 4/C412), the operation of the work attachment 440 and the like is controlled (FIG. 4/STEP 412). As a result, for example, the magnetic force generated by the electromagnet, which is the end effector 445, attracts iron scraps piled up at a predetermined location, and after the upper rotating body 420 is rotated, the electromagnet as the end effector 445 generates magnetic force. By stopping the end effector 445, the operation of dropping the iron scrap onto the loading platform 52 of the dump truck 50 is executed.

(Second function)
The second function of the remote operation support system configured as described above will be described with reference to the flow chart shown in FIG. Also in the flowchart, the block "C" is used for the sake of simplification of description, and means transmission and/or reception of data, and processing in the branch direction is executed on the condition of transmission and/or reception of the data. It means a conditional branch that

In work machine 40, actual machine control device 400 measures the first index value and the second index value using state sensor 414 constituting actual machine input interface 41 (FIG. 5/STEP 420). The "first index value" is an index value representing the distance between the upper swing body 420 and the end effector 445. And, based on the output signal of the state sensor 414 representing the rotation angle of the end effector 445 with respect to the tip of the arm 443, it is measured according to forward kinematics. “Second index value” is an index value representing the turning angle of upper turning body 420 with respect to lower traveling body 410, and is measured based on the output signal of state sensor 414 representing the turning angle or turning angle of turning mechanism 430. be.

The actual device control device 400 uses the actual device wireless communication device 422 to transmit the respective measured values of the first index value and the second index value or the current measured value to the remote operation support server 10 (FIG. 5/STEP 422 ).

When the remote operation support server 100 receives the measured values of the first index value and the second index value (FIG. 5/C120), the first support processing element 121 transmits an indicator output command (FIG. 5/C120). STEP 120). The “indicator output command” is a command for causing the image output device 221 of the remote control device 20 to output an indicator representing the measured values of the first index value and the second index value.

In the remote control device 20, when the indicator output command is received by the remote wireless communication device 222 (FIG. 5/C220), the remote control device 200 outputs the first indicator M1 representing the first index value and the second indicator M1 representing the second index value. 2 indicator M2 is output and displayed on the image output device 221 (FIG. 5/STEP 220).

In the remote operation device 20, the presence or absence of the first designated operation through the remote input interface 210 is determined by the operator (FIG. 4/STEP 221). The "first designated operation" is, for example, an operation of the remote control mechanism 211 that constitutes the remote input interface 210 for starting interaction between the end effector 445 and an object (adsorbing an object such as iron scrap). If the determination result is negative (FIG. 4/STEP 221 . . . NO), the processing after the indicator is output is repeated. On the other hand, if the determination result is affirmative (FIG. 4/STEP 221 . . . YES), the remote operation support server 10 issues the first operation command for starting the interaction between the end effector 445 and the object through the remote wireless communication device 222 . (FIG. 4/STEP 222).

In the remote operation support server 10, when the first operation command is received by the first support processing element 121, the first operation command is transmitted to the work machine 40 by the second support processing element 122 (see FIG. 4). /C121).

In the work machine 40, when the actual machine controller 400 receives the first operation command through the actual machine wireless communication device 422 (FIG. 4/C421), the electromagnet as the end effector 445 generates a magnetic force (FIG. 4/STEP 424). ). As a result, for example, iron scraps piled up at a predetermined location are attracted by the magnetic force generated by the electromagnet, which is the end effector 445 .

Further, in the remote operation support server 10, when the first operation command is received by the first support processing element 121 (FIG. 4/C121), the second support processing element 122 outputs the first output command to the remote control device 20. (FIG. 4/STEP 121). The "first output command" is for causing the image output device 221 of the remote control device 20 to output the indicators representing the measured values of the first index value and the second index value so that the indicators represent the first reference value. is a directive.

In the remote control device 20, when the first output command is received by the remote wireless communication device 222 (FIG. 5/C221), the remote control device 200 outputs the first index value at the first reference point in time when the first designated operation is performed. and at least one of the second indicator M2 representing the second index value is output and displayed on the image output device 223 (FIG. 5/STEP 223).

In the remote operation device 20, it is determined whether or not the operator has performed the second designated operation through the remote input interface 210 (FIG. 4/STEP 224). The "second specified operation" is, for example, an operation of the remote control mechanism 211 that constitutes the remote input interface 210 for ending the interaction between the end effector 445 and the object (releasing the object such as iron scrap). If the determination result is negative (FIG. 4/STEP 224 . . . NO), the processing after the indicator is output is repeated. On the other hand, if the determination result is affirmative (FIG. 4/STEP 224 . . . YES), the remote operation support server 10 issues a second operation command to end the interaction between the end effector 445 and the object through the remote wireless communication device 222 . (FIG. 4/STEP 225).

In the remote operation support server 10, when the first support processing element 121 receives the second operation command, the second support processing element 122 transmits the second operation command to the work machine 40 (FIG. 4). /C122).

In the work machine 40, when the actual machine control device 400 receives the second operation command through the actual machine wireless communication device 422 (Fig. 4/C422), the electromagnet as the end effector 445 stops generating the magnetic force (Fig. 4 / STEP 426). As a result, the iron scrap is released by stopping the magnetic force generated by the electromagnet, which is the end effector 445, for example.

Further, in the remote operation support server 10, when the first support processing element 121 receives the second operation command (FIG. 4/C121), the second support processing element 122 outputs the second output command to the remote control device 20. (FIG. 4/STEP 122). The "second output command" is for causing the image output device 221 of the remote control device 20 to output the indicators representing the measured values of the first index value and the second index value so that the indicators represent the second reference value. is a directive.

In the remote control device 20, when the second output command is received by the remote wireless communication device 222 (FIG. 5/C222), the remote control device 200 outputs the first index value at the second reference time when the second designated operation is performed. and at least one of the first indicator M1 representing the second index value and the second indicator M2 representing the second index value are output and displayed on the image output device 223 (FIG. 5/STEP 226).

(Display mode of the first indicator)
For example, in the first work state S11 shown on the left side of FIG. 6A, the closer the first index value is to its lower limit, the closer the upper rotating body 420 and the end effector 445 are. In the first working state S11, as shown on the left side of FIG. 6B, the first indicator M1 has its inner peripheral edge aligned with the inner circle M12 representing the upper limit of the first index value, and its outer peripheral edge is represented by a substantially annular figure approaching the outer circle M11 representing the lower limit.

In the second working state S12 shown in the center of FIG. 6A, the first index value is the first point in time when the end effector 445 starts interacting with an object (adsorbs an object such as scrap metal), and/or , represents the distance between the upper rotating body 420 and the end effector 445 at the second point in time when the end effector 445 has finished interacting with the object (released the object such as iron scrap). In the second working state S12, as shown in the center of FIG. 6B, the first indicator M1 has its inner peripheral edge aligned with the inner circle M12 representing the lower limit of the first index value, and its outer peripheral edge is represented by a substantially circular figure approaching the intermediate circle M10 representing the distance (first reference distance and/or second reference distance).

In the third work state S13 shown on the right side of FIG. 6A, the upper revolving body 420 and the end effector 445 are separated as the first index value approaches its upper limit value. In the third working state S13, as shown on the right side of FIG. 6B, the first indicator M1 has its inner peripheral edge aligned with the inner circle M12 representing the upper limit of the first index value, and its outer peripheral edge is represented by a substantially annular figure approaching the inner circle M12.

(Display mode of the second indicator)
For example, in the first turning state S21 shown at the left end of FIG. 7A, the second index value is the lower part at the second point in time when the end effector 445 has finished interacting with the object (released the object such as scrap metal). It represents the turning angle of the upper turning body 420 with respect to the traveling body 410 . In the first turning state S21, as shown in the left end of FIG. 7B, the second indicator M2 is represented by a rectangle whose both ends coincide with the gauge M22 representing the second reference value of the second index value. (ie, the rectangle is not displayed).

In the second turning state S22 shown on the center left side of FIG. 7A, the second index value is clockwise from the second point in time when the end effector 445 has finished interacting with the object (released the object such as iron scrap). Alternatively, it represents the turning angle of the upper turning body 420 with respect to the lower traveling body 410 in a clockwise turning state. In the second turning state S22, as shown in the center left of FIG. 7B, the second indicator M2 is represented by a rectangle whose both ends coincide with the gauge M22 representing the second reference value of the second index value. (ie, the rectangle is not displayed).

In the third turning state S23 shown on the center right side of FIG. 7A, the second index value indicates that the end effector 445 is traveling under the lower part at the first point in time when the end effector 445 starts interacting with an object (adsorbing an object such as iron scrap). It represents the turning angle of the upper turning body 420 with respect to the turning body 410 . In the third turning state S23, as shown in the center right of FIG. 7B, one end of the second indicator M2 coincides with the gauge M22 representing the second reference value of the second index value, and the second indicator M2 It is represented by a rectangle coinciding with the gauge M21 representing the first reference value of the index value.

In the fourth turning state S24 shown at the left end of FIG. 7A, the second index value is counterclockwise from the first point in time when the end effector 445 starts interacting with an object (adsorbs an object such as iron scrap). Alternatively, it represents the turning angle of the upper turning body 420 with respect to the lower traveling body 410 in a state of turning counterclockwise. In the second turning state S22, as shown on the left side of FIG. 7B, one end of the second indicator M2 coincides with the gauge M22 representing the second reference value of the second index value, and the second index It is represented by a rectangle spaced from the gauge M21 representing the first reference value of the value toward the gauge M22.

The second index value rotates counterclockwise or counterclockwise from the first point in time when the end effector 445 starts interacting with the object (adsorbs an object such as iron scrap), and the end effector 445 stops interacting with the object. When the second point in time (overrunning) of the last time (at which an object such as iron scrap was sucked) is exceeded, the second indicator M2 has one end corresponding to the gauge M22 representing the reference value of the second index value. , and is represented by a rectangle spaced toward the gauge M20 on the opposite side of the gauge M21 representing the second reference value of the second index value.

(Example)
The state shown in FIG. 8A corresponds to the second working state S12 (see center of FIG. 6A) and the first turning state S21 (see left end of FIG. 7A). Therefore, in this state, as shown in FIG. 8B, the first indicator M1 (see the center of FIG. 6B) corresponding to the second work state S12 superimposed on the work environment image and the first indicator M1 corresponding to the first turning state S21 are displayed. A second indicator M2 (see the left end of FIG. 7B) is displayed.

The state shown in FIG. 9A corresponds to the second working state S12 (see center of FIG. 6A) and the second turning state S22 (see center left of FIG. 7A). Therefore, in this state, as shown in FIG. 9B, the first indicator M1 (see the center of FIG. 6B) corresponding to the second work state S12 superimposed on the work environment image and the second indicator M1 corresponding to the second turning state S22. A second indicator M2 (see the center left side of FIG. 7B) is displayed.

The state shown in FIG. 10A corresponds to the second working state S12 (see center of FIG. 6A) and the third turning state S23 (see center right of FIG. 7A). Therefore, in this state, as shown in FIG. 10B, the first indicator M1 (see the center of FIG. 6B) corresponding to the second work state S12 superimposed on the work environment image and the third indicator M1 corresponding to the third turning state S23. A second indicator M2 (see the center right side of FIG. 7B) is displayed.

The state shown in FIG. 11A corresponds to the second working state S12 (see center of FIG. 6A) and fourth turning state S24 (see left end of FIG. 7A). Therefore, in this state, as shown in FIG. 11B, the first indicator M1 (see the center of FIG. 6B) corresponding to the second work state S12 superimposed on the work environment image and the fourth turning state S24 are displayed. A second indicator M2 (see the left end of FIG. 7B) is displayed.

The state shown in FIG. 12A corresponds to the first work state S11 (see the left side of FIG. 6A) and the first turning state S21 (see the left end of FIG. 7A). Therefore, in this state, as shown in FIG. 12B, the first indicator M1 (see the left side of FIG. 6B) corresponding to the first work state S11 is superimposed on the work environment image and the first indicator M1 corresponding to the first turning state S21. A second indicator M2 (see the left end of FIG. 7B) is displayed.

The state shown in FIG. 13A corresponds to the third working state S13 (see the right side of FIG. 6A) and the first turning state S21 (see the left end of FIG. 7A). Therefore, in this state, as shown in FIG. 13B, the first indicator M1 (see the right side of FIG. 6B) corresponding to the second work state S12 superimposed on the work environment image and the first indicator M1 corresponding to the first turning state S21. A second indicator M2 (see the left end of FIG. 7B) is displayed.

(effect)
According to the remote operation support system having this configuration and the remote operation support server 10 that configures it, the work machine can be operated through the indicators M1 and M2 that are output to the remote output interface 220 (image output device 221) that configures the remote control device 20. The operator of the remote control device 20 can intuitively recognize the distance between the end effector 445 of 40 and the upper rotating body 420 and/or the turning angle of the upper rotating body 420 with respect to the lower traveling body 410 (Fig. 6B). , 7B, 8B, 9B, 10B, 11B, 12B and 13B). Also, through the time change of the indicators M1 and M2, the length of the distance between the end effector 445 of the work machine 40 and the upper revolving body 21 and/or the time change of the revolving angle of the upper revolving body 420 with respect to the lower traveling body 410 can be intuitively understood. 8B, 9B, 10B, 11B, 12B and 13B).

The index value at the start time (first time point) of the interaction between the end effector 445 and the object (for example, iron scrap) is recognized as the first reference index value, the first indicator M1 represents the first reference index value M11, In addition, the indicators M1 and M2 are output to the image output device 221 so that the second indicator M2 represents the second index reference value M21 (see the center of FIG. 6B and the right side of the center of FIG. 7B). The index value at the end point (second point in time) of the interaction between the end effector 445 and the object (for example, iron scrap) is recognized as the second reference index value, the first indicator M1 represents the second reference index value M12, In addition, the indicators M1 and M2 are output to the image output device 221 so that the second indicator M2 represents the second index reference value M22 (see the center of FIG. 6B and the left end of FIG. 7B).

By comparing the indicators M1 and M2 representing the current index values with the indicators M1 and M2 representing the reference index values M11, M12, M21 and M22 output to the remote output interface 220 (image output device 221) as a reference, The distance between the end effector 445 and the upper rotating body 420 and/or the upper rotating body 410 with respect to the current distance between the end effector 445 and the upper rotating body 410 is determined based on at least one of the start time and the end time of the interaction between the end effector 445 and the object. The deviation of the turning angle of the body 420 can be intuitively recognized by the operator of the remote control device 20 .

Reference gauges M11 and M12 representing at least one of the upper limit value and the lower limit value of first index value M1 are output to remote output interface 220 (image output device 221).

By comparing the first indicator M1 and the reference gauges M11, M12 output to the remote output interface 220 (image output device 221), the first index value represented by the first indicator M1 is determined to be the upper limit value and/or the lower limit value. The operator of the remote control device 20 can intuitively recognize how close the values are (see left and right sides of FIG. 6B).

10 -- remote operation support server, 20 -- remote control device, 40 -- work machine, 41 -- actual machine input interface, 42 -- actual machine output interface, 102 -- database, 121 -- first support processing element, 122 -- second support processing element, 200 Remote control device 210 Remote input interface 211 Remote control mechanism 220 Remote output interface 221 Image output device 400 Actual machine control device 410 Lower traveling body 420 Upper revolving body 424 A cab (driver's cab), 440... work attachment (actuating mechanism), 445... a magnet (end effector).

Claims (6)

To support remote operation using a remote control device of a working machine having a lower traveling body, an upper revolving body that can swivel with respect to the lower traveling body, and a work attachment capable of moving an end effector with respect to the upper revolving body. A remote operation support server of
a first support processing element that recognizes an index value representing a distance between the upper rotating body and the end effector based on communication with the work machine;
a second support processing element for outputting an indicator representing the index value recognized by the first support processing element to an output interface constituting the remote control device based on communication with the remote control device; A remote operation support server. The remote operation support server according to claim 1,
The remote operation support server, wherein the first support processing element recognizes the index value representing the swing angle of the upper swing body with respect to the undercarriage based on communication with the work machine. In the remote operation support server according to claim 1 or 2 ,
The first support processing element recognizes, as a reference index value, the index value at least one of a start point and an end point of interaction between the end effector and the object based on communication with the work machine. A remote operation support server. In the remote operation support server according to any one of claims 1 to 3 ,
The remote operation support server, wherein the second support processing element causes the output interface to output a reference gauge representing at least one of an upper limit value and a lower limit value of the index value together with the indicator. A remote operation support system, comprising: the remote operation support server according to any one of claims 1 to 4 ; and the remote control device. To support remote operation using a remote control device of a working machine having a lower traveling body, an upper revolving body that can swivel with respect to the lower traveling body, and a work attachment capable of moving an end effector with respect to the upper revolving body. The remote operation support method of
a first support process for recognizing an index value representing a distance between the upper revolving body and the end effector based on communication with the work machine;
and executing a second support process for outputting an indicator representing the index value recognized by the first support process to an output interface constituting the remote control device based on communication with the remote control device. A remote operation support method characterized by:

Images