EP3926107A1

EP3926107A1 - A system, an off-board controller and method therein for remote operation of a working machine comprising a tool

Info

Publication number: EP3926107A1
Application number: EP20180599.1A
Authority: EP
Inventors: Calle SKILLSÄTER; Mikael FRIES
Original assignee: Volvo Construction Equipment AB
Current assignee: Volvo Construction Equipment AB
Priority date: 2020-06-17
Filing date: 2020-06-17
Publication date: 2021-12-22

Abstract

A system (10) for remote operation of a working machine (110) comprising a tool (111) is provided. The system comprises a remote control station (5) comprising a work station (102), at least one display (11), and an off-board controller (100). The system also comprises an on-board controller (114) for arrangement on the working machine and arranged to receive signals from the remote control station (5) remotely controlling the operation of the working machine, wherein the on-board controller is arranged to obtain sensor data from at least one sensor on the working machine and camera images from at least one camera (112, 113) on the working machine. The off-board controller is further arranged to: receive, via a communication interface (205), sensor data and camera images from the on-board controller (114) on the working machine (110); analyse the sensor data to determine a location of the tool (111) in relation to the surroundings of the working machine (110); determine at least one overlay (410, 420) indicating the location of the tool (111) in relation to surroundings of the working machine; and command display of the camera images together with the at least one determined overlay (410, 420) on the at least one display (11) for supporting the operator in determining the location of the tool in relation to the surroundings of the working machine.An off-board controller (100) for remote operation of a working machine (110) and method therein are also provided.

Description

TECHNICAL FIELD

Embodiments herein relate in general to remote operation of a working machine. In particular, embodiments herein relate to a system for remote operation of a working machine comprising a tool. Also, the embodiments herein also relate to an off-board controller and a method therein for remote operation of a working machine comprising a tool.

BACKGROUND

There is a high level of complexity in setting up a remote operation of a working machine, such as, e.g. an excavator, fork lift, loader, hauler, etc., that allows an operator to control the working machine from a remote location, such as, e.g. from a control room, instead of from a driver seat of the working machine. Specifically, it is often difficult to provide the operator with a similar sense and experience of the on-premise surroundings of the working machine as the operator would have when, for example, sitting in the driver seat of the working machine overlooking and having a good view of the surroundings.
In remote operation, the operator may typically control the machine from the remote location by looking onto one or more screens that are visualizing the surroundings of working machine via one or more image/video feeds from one or more cameras arranged on the working machine. One problem with this configuration is that it is not able to provide any real depth perception, since the image/video feeds provided is typically not able to visualize anything else than standard 2D images. This makes it hard to, for example, remotely operate a loader comprising a bucket for lifting and moving a pile of material since it will be difficult for the operator to see and determine where the bucket will actually enter the pile. Hence, it will be hard for an operator to control the working machine in the most suitable way which will directly impact the productivity of the working machine.
Hence, there is need to be able to increase the sense and experience of the surroundings of the working machine to an operator operating the working machine from a remote location.

SUMMARY

It is an object of embodiments herein to improve remote operation of a working machine.
According to a first aspect of embodiments herein, the object is achieved by a system for remote operation of a working machine comprising a tool. The system comprises a remote control station comprising a work station, at least one display, and an off-board controller. The system further comprises an on-board controller for arrangement on the working machine and arranged to receive signals from the remote control station remotely controlling the operation of the working machine. The on-board controller is arranged to obtain sensor data from at least one sensor on the working machine and camera images from at least one camera on the working machine. Furthermore, the off-board controller is arranged to receive, via a communication interface, sensor data and camera images from the on-board controller on the working machine. Further, the off-board controller is arranged to analyse the sensor data to determine a location of the tool in relation to the surroundings of the working machine, determine at least one overlay indicating the location of the tool in relation to surroundings of the working machine, and command display of the camera images together with the at least one determined overlay on the at least one display for supporting the operator in determining the location of the tool in relation to the surroundings of the working machine.
By providing a system as described above, the sense and experience of the surroundings of a working machine for an operator located at a remote location is enhanced by the operator being provided with one or more augmented and visual overlays in the displayed image/video feed that takes both the working machine and any attached tool into consideration. Hence, remote operation of the working machine is improved.
According to some embodiments, the sensor data may comprise at least one of a tool height over ground, a tool articulation in relation to a body of the working machine, a traveling direction of the working machine, and a distance between the tool and a target contact position. Thus, the off-board controller of the system may be provided with different types of information relating to the operation and surroundings of the working machine and the tool. In some embodiments, the system may also comprise a tool identification sensor for location on the working machine, wherein the off-board controller is arranged to receive information about a tool type from the tool identification sensor. This means that the form of the at least one determined overlay may be adapted based on the tool that is currently mounted on the working machine without any additional information being required. This also enables the off-board controller to automatically adapt the form of the at least one determined overlay as a tool is exchanged for another on the working machine. In some embodiments, the tool identification sensor may comprise a radio frequency identification, RFID, sensor, and wherein the tool comprises an RFID tag comprising information about the tool type.
Further, the off-board controller may according to some embodiments be arranged to determine the at least one determined overlay by means of image analysis of the camera images. Also, here, the off-board controller may be further arranged to provide overlay information related to identification of the tool and tool dimensions in the camera images. This advantageously enables received and analysed sensor data to be used together with images analysis of the camera images in order to, for example, be able to determine an accurate relative size and position of a virtual shadow of the tool in the camera images.
In some embodiments, the at least one determined overlay may comprise a virtual shadow on the ground under the tool for indicating the location of the tool. This may advantageously provide an indicative depth perception to the remote operator and location of the tool relative to the surroundings of the working machine. Optionally, the at least one determined overlay may also comprise an indication of a distance above the ground of the tool or a distance between the tool and a target contact position. This may provide a visual feedback of how the tool and vehicle relate to certain parts of the surroundings. In some embodiments, the system may be arranged to determine the distance above the ground or distance to the target contact position using at least one of LIDAR, radar, ultrasound sensor, angle sensors, inertial measurement unit, or image analysis based on stereo vision images. Further, according to some embodiments, the indication of distance above ground or distance between tool and target contact position comprises a colour indication or a texture indication. This advantageously provides a more intuitive and easy accessible visual feedback to the remote operator. According to another option, the at least one determined overlay may also comprise an interpolated path of travel in relation to a current travel path. This advantageously allows the remote operator to get an improved visual feedback on how the working machine and the mounted tool relates to the surroundings when the working machine is in motion.
According to some embodiments, the off-board controller of the system may further be arranged to determine the location of the tool in relation to surroundings of the working machine by determining the location of the tool in relation to a ground below the tool, to a ground adjacent to the working machine, and/or to the working machine. This information may increase the possibility of the off-board controller to provide even more accurate and detailed visual feedback via the displayed camera images and at least one determined overlay.
According to a second aspect of embodiments herein, the object is achieved by an off-board controller for remote operation of a working machine comprising a tool. The off-board controller comprises a processing device, a computer readable storage unit, a communication interface, and a display interface. The processing device is arranged to execute instruction sets stored in the storage unit in order to: receive, via the communication interface, sensor data and camera images from an on-board controller on the working machine; analyse the sensor data for determining a location of the tool in relation to the surroundings of the working machine; determine at least one overlay indicating the location of the tool in relation to the surroundings; and command display of, via the display interface, camera images together with the at least one determined overlay on a display of a remote control station to support the operator in determining the location of the tool in relation to the surroundings of the working machine.
By providing an off-board controller as described above, the sense and experience of the surroundings of a working machine for an operator located at a remote location is enhanced by being provided with one or more augmented and visual overlays in the displayed image/video feed that takes both the working machine and any attached tool into consideration. Hence, remote operation of the working machine is improved.
In some embodiments, the at least one determined overlay may comprise at least one of: a virtual shadow on the ground under the tool for indicating the location of the tool; an overlay shape over the tool with a colour, where the colour indicates a range to ground or range to a target contact position; and a future direction of travel of the working machine related to current direction of travel of the working machine. Each of these determined overlays enables a remote operator that is observing the camera images comprising the determined overlays to get a better sense and experience of the surroundings of the working machine when operating the working machine from a remote location.
According to some embodiments, the at least one determined overlay may comprise a virtual shadow on the ground, and wherein the virtual shadow of the ground is displayed in a colour indicating a range between the tool and ground. This provides additional sense and experience of the surroundings of the working machine in that the range may be color-coded in the determined overlay.
According to a third aspect of the embodiments herein, the object is achieved by a method performed by an off-board controller for remote operation of a working machine. The method comprises the steps of: receiving, by an off-board controller, sensor data associated with the working machine and with a tool of the working machine, and camera images from an on-board controller on the working machine; analysing the sensor data to determine a location of the tool in relation to the surroundings of the working machine; determining at least one overlay indicating the location of the tool in relation to the surroundings; and displaying the camera images together with the determined at least one determined overlay on a display in a control station for supporting the operator in determining the location of the tool in relation to the surroundings of the working machine.
By providing an method as described above, the sense and experience of the surroundings of a working machine for an operator located at a remote location is enhanced by being provided with one or more augmented and visual overlays in the displayed image/video feed that takes both the working machine and any attached tool into consideration. Hence, remote operation of the working machine is improved
In some embodiments, the at least one determined overlay may comprise at least one of: a virtual shadow on the ground under the tool for indicating the location of the tool; an overlay shape over the tool with a colour, where the colour indicates a range to ground or range to a target contact position; and a future direction of travel of the working machine interpolated from current direction of travel of the working machine. As indicated above, each of these determined overlays enables a remote operator that is observing the camera images comprising the determined overlays to get a better sense and experience of the surroundings of the working machine when operating the working machine from a remote location. Further, in some embodiments, the at least one determined overlay may comprise a virtual shadow on the ground, and wherein the virtual shadow of the ground is displayed in a colour indicating a range between the tool and ground. This provides additional sense and experience of the surroundings of the working machine in that the range may be color-coded in the determined overlay.
According to a fourth aspect of the embodiments herein, the object is achieved by a computer program comprising program code means for performing the steps of the method described above when said computer program is run on a computer. According to a fifth aspect of the embodiments herein, the object is achieved by a computer readable medium carrying a computer program comprising program code means for performing the steps of the method described above when said computer program is run on a computer. In some embodiments, the computer readable medium may be one of an electronic signal, optical signal, radio signal, or computer-readable storage medium.

BRIEF DESCRIPTION OF THE DRAWINGS

Features and advantages of the embodiments will become readily apparent to those skilled in the art by the following detailed description of exemplary embodiments thereof with reference to the accompanying drawings, wherein:

Fig. 1: is a schematic illustration of a working machine and a system for remote operation of the working machine according to some embodiments,
Fig. 2: is a flowchart illustrating embodiments of a method in an off-board controller,
Fig. 3: is a schematic view of a working machine according to some embodiments,
Fig. 4: is a schematic view of a display of camera images according to some embodiments,
Fig. 5: is a schematic view of an off-board controller according to some embodiments, and
Fig. 6: is a schematic view of an on-board controller according to some embodiments.

DETAILED DESCRIPTION

The figures are schematic and simplified for clarity, and they merely show details which are essential to the understanding of the embodiments presented herein, while other details have been left out. Throughout, the same reference numerals are used for identical or corresponding parts or steps.
Fig. 1 shows a system 10 for remote operation of a working machine 110 comprising a tool 111 according to some embodiments. The working machine 110 may for example, be an excavator, fork lift, loader, hauler, etc., or any other working machine onto which a working tool may be arranged and operated. The tool 111 may be a large variety of different working tools, such as, e.g. loader buckets, buckets of different sizes, fork lifts, etc. The tools 111 may be arranged to be connected or attached to the working machine 110 and operated therefrom. In the example illustrated in Fig. 1, the working machine 110 is a loader comprising a tool 111 in the form of a bucket to, e.g. move a pile of loose material 150.
As part of the system 10, the working machine 110 also comprise an on-board controller 114, at least one camera 112, 113, at least one sensor 116 and at least one antenna 115. The on-board controller 114 may be connected to and arranged to communicate with the at least one camera 112, 113 and the at least one sensor 116, e.g. wirelessly or by wire. The on-board controller 114 may also be connected to and arranged to communicate a Global Positioning System, GPS, unit (not shown) in the working machine 110. Via the antenna 115, the on-board controller 114 may receive signals remotely from an off-board controller 100. Thus, the on-board controller 114 is adapted to be arranged on the working machine 110 and arranged to receive signals from a remote control station 5 remotely controlling the operation of the working machine 110. The on-board controller 114 is arranged to obtain sensor data from at least one sensor 116 on the working machine 110 and camera images from at least one camera 112, 113 on the working machine 110.
Further, also as part of the system 10, an off-board controller 100 may be connected via a communication link 120 to a communications server 106. The communications server 106 may be part of a wireless communications network 105 arranged for wireless communication over an air interface with the antenna 115 and on-board controller 114. The air interface may, for example, be provided by a telecommunications network, such as, e.g. a 3g/4g/LTE/5g network or other future networks, and/or by a local wireless network, such as, e.g. WiFi, etc. This allows the off-board controller 100 to communicate with the on-board controller 114 located on the working machine 110. The off-board controller 100 may also be arranged in, or connected to, a remote control station 5. The remote control station 5 may comprise a work station 102 and at least one display 11. The off-board controller 100 may be arranged to communicate with the work station 102 and the at least one display 11 over one or more communication links 130. The off-board controller 100 of the system 10 is described below in more detail in reference to Figs. 2-5 and the on-board controller 114 of the system 10 is described below in more detail in reference to Fig. 6.
Examples of embodiments of a method performed by an off-board controller 100 for remote operation of a working machine 110 comprising a tool 111, will now be described with reference to the flowchart depicted in Fig. 2. Fig. 2 is an illustrated example of actions, steps or operations which may be taken by the off-board controller 100 in the system 10. The method may comprise the following actions, steps or operations.

Action 201

The off-board controller 100 receives sensor data and camera images from an on-board controller 114 on the working machine 110. According to some embodiments, the sensor data comprises at least one of: a tool height over ground, a tool articulation in relation to a body of the working machine 110, a traveling direction of the working machine 110, and a distance between the tool 111 and a target contact position 150. This means that the off-board controller 100 may be provided with information different types of information regarding the operation and surroundings of the working machine 110, such as, e.g. articulation/steering angles of the working machine 110, etc. Also, the sensor data may also comprise characteristics of the working machine 110, such as, e.g. machine steering characteristics.

Action 202

After receiving the sensor data in Action 201, the off-board controller 100 analyses the sensor data to determine a location of the tool 111 in relation to the surroundings of the working machine 110. Here, in some embodiments, the off-board controller 100 may be further arranged to determine the location of the tool 111 in relation to surroundings of the working machine 110 by determining the location of the tool 111 in relation to a ground below the tool, to a ground adjacent to the working machine 110 and/or to the working machine 110.
In some embodiments, the off-board controller 100 may be arranged to receive information about a tool type from a tool identification sensor 116. The information about the tool type may provide further information about the tool 111, such as, e.g. tool width. For example, via a look-up table, the mounted tool identity may be matched to a specific tool width. Here, the at least one sensor 116 in the system 10 may comprise a tool identification sensor 116 located on the working machine 110. In some embodiments, the tool identification sensor 116 may comprise a radio frequency identification, RFID, sensor, and wherein the tool comprises an RFID tag comprising information about the tool type. This means that the form of the at least one overlay 410, 420 determined in Action 203 may be adapted based on the mounted tool 111 on the working machine 110. In other words, the overlay 410, 420 may automatically change if the tool 111 is changed on the working machine 110, e.g. changing between buckets of different size or from a bucket to fork lift, etc.

Action 203

The off-board controller 100 determine at least one overlay 410, 420 indicating the location of the tool 111 in relation to the surroundings of the working machine 110. In some embodiments, the at least one determined overlay 410, 420 may comprise at least one of: a virtual shadow on the ground under the tool 111 for indicating the location of the tool 111, an overlay shape over the tool with a colour, where the colour indicates a range to ground or range to a target contact position, and a future direction of travel of the working machine interpolated from current direction of travel of the working machine. This means that, for example, that augmented or visual overlays 410, 420 may be obtained which may then be inserted into the received camera images to provide support for the remote operator in remote operation of the working machine 110.
Further, in some embodiments, the off-board controller 100 may be arranged to determine the at least one determined overlay 410, 420 by means of image analysis of the camera images. In this case, the off-board controller 100 may according to some embodiments further be arranged to provide overlay information related to identification of the tool 111 and tool dimensions in the camera images. This means, for example, the received and analysed sensor data in Action 201 may be used to together with images analysis of the camera images in order to, for example, determine a relative size and position of the virtual shadow of the tool 11 in the camera images. This so as to provide an indicative depth perception to the remote operator.
According to some embodiments, the at least one determined overlay 410, 420 may comprise an indication of a distance above the ground of the tool 111 or a distance between the tool 111 and a target contact position 150. Further, in some embodiments, the indication of distance above ground or distance between tool 111 and target contact position 150 may comprise, for example, a colour indication or a texture indication. This means, for example, that the at least one overlay 410, 420 may be colour coded based on the distance from a bucket to a pile, i.e. from the tool 111 to the target contact position 150, i.e. the object. As an example, the at least one overlay 410, 420 may be green when the pile or object is far away from the tool 111 and change to red when the tool 111 is getting closer to the pile or object.
The off-board controller 100 may also be arranged to determine the distance above the ground or distance to the target contact position 150 using at least one of LIDAR, radar, ultrasound sensor, angle sensors, inertial measurement unit, or image analysis based on stereo vision images. Thus, the at least one sensor 116 in the system 10 may comprise at least one of LIDAR, radar, ultrasound sensor, angle sensors, inertial measurement unit, or image analysis unit. In some embodiments, the hydraulic sensors of the working machine 110 may also be used to determine the distance above the ground or distance to the target contact position 150.
Fig. 3 illustrates how at least one sensor 116 in the system 10 may be used to determine a steering angle 300 between the working machine 110 and the tool 111, e.g. by using an angle sensor. This steering angle 300 may then be used in the determination of the at least one overlay 410, 420, such as, e.g. an illustrated driving path of the working machine 110.

Action 204

After determining the at least one overlay in Action 203, the off-board controller 100 displays the camera images together with the determined at least one determined overlay 410, 420 on a display 11 in a control station 5 for supporting the operator in determining the location of the tool 111 in relation to the surroundings of the working machine 110. This means, for example, that a remote operator of a working machine 110 may be provided with an illustrated driving path and virtual shadow of the tool 11, via the visual overlay 410, 420 in the camera images, on the display 11 in the remote control station 5 that takes both the working machine 110 and the tool 111 into account. Here, the received and analysed sensor data may, for example, be used to interpolate the driving path of the working machine 110, e.g. determining where a tool 111, such as, a bucket, will enter a pile 150 based on sensor data, such as, current articulation/steering angle, etc., or where a working machine 110 will drive on a road segment. Also, the received and analysed sensor data may, for example, be used to determine a relative size and position of the virtual shadow of the tool 11 in the camera images so as to provide an indicative depth perception for the remote operator. The visual overlay 410, 420 may then be displayed to the remote operator through the camera images, i.e. received real-time video stream or video feed from the cameras on the working machine 110, via the display 11 of the remote control station 5.
Fig. 4 illustrates an example of a display 11 of the camera images 400 according to some embodiments. The camera images 400 in Fig. 3 comprise a partial view of the working machine 110 and a view of the tool 111 of the working machine 110. Also, the camera images 400 in Fig. 4 also display the at least one overlay 410, 420 and the target contact position 150, which e.g. may be a pile of loose material 150 as shown in Fig. 1.
In this example, cameras mounted on the remotely operated working machine 110 send an image/video feed to an off-board controller 100 at a remote control station 5 via a wireless interface. In the off-board controller 100, an augmented visual overlay 410, 420 may be inserted into the image/video stream in order to, for example, illustrate to the remote operator in the remote control station 5 where the working machine 110 is heading. The augmented visual overlay 410, 420 may here be based e.g. articulation (e.g. the steering angle 300 in Fig. 3) and/or other sensor data and information provided by an on-board controller 114 from the working machine 110. The on-board controller 114 of the working machine 110 may collect further sensor data and information about the working machine 110, and also, if applicable, collect further sensor data and information about the mounted tool 111 attached to the working machine 110. This sensor data and information may comprise, e.g. information about the width of the tool 111 and/or the working machine 110, the articulation/steering angle of the working machine 110 and related machine characteristics. The on-board controller 114 may then send this data and information to the off-board controller 110. In the off-board controller 100, the augmented visual overlay 410, 420 illustrating the future driving path of the working machine 110 may be calculated based on the collected sensor data and information from the on-board controller 114. This augmented visual overlay 410, 420 may then be inserted in real-time onto the real-time image/video stream or feed from the working machine 110 and presented on the display 11. In this example, this is illustrated in Fig. 4 as a "virtual shadow" 320 under the bucket 111 indicating the distance between the bucket 111 and the ground, as well as, illustrated or virtual lines 310 indicating the heading of the working machine 110.
To perform the method actions for remote operation of a working machine 110 comprising a tool 111, the system 10 may comprise an off-board controller 100 having the following arrangement depicted in Fig. 5. Fig. 5 shows a schematic view of an off-board controller 100 according to some embodiments. The on-board controller 100 may comprise a processing circuitry or device 501, a computer readable storage unit 502, a display interface 503, and a communication interface 504. The processing circuitry 501 may be arranged to execute instruction sets stored in the computer readable storage unit 502.
The off-board controller 100 or processing circuitry 501 is configured to, or may comprise a receiving module 550 configured to, receive, e.g. via a communication interface 504, sensor data and camera images from the on-board controller 114 on the working machine 110. The off-board controller 100 or processing circuitry 501 is also configured to, or may comprise an analyzing module 560 configured to, analyse the sensor data to determine a location of the tool 111 in relation to a ground below the tool 111 and to the working machine 110. The off-board controller 100 or processing circuitry 501 is further configured to, or may comprise a determining module 570 configured to, determine at least one overlay indicating the location of the tool 111 in relation to surroundings of the working machine 110. Furthermore, the off-board controller 100 or processing circuitry 501 is further configured to, or may comprise a commanding module 580 configured to, command display, e.g. via the display interface 503, of the camera images together with the at least one determined overlay on the at least one display 11 for supporting the operator in determining the location of the tool 111 in relation to the surroundings of the working machine 110.
Furthermore, the embodiments for remote operation of a working machine 110 comprising a tool 111 described above may be at least partly implemented through one or more processors, such as the processing circuitry 501 in the off-board controller 100 depicted in Fig. 5, together with computer program code for performing the functions and actions of the embodiments herein. The program code mentioned above may also be provided as a computer program product, for instance in the form of a data carrier carrying computer program code or code means for performing the embodiments herein when being loaded into the processing circuitry 501 in the off-board controller 100. The data carrier, or computer readable medium, may be one of an electronic signal, optical signal, radio signal, or computer-readable storage medium. The computer program code may e.g. be provided as pure program code in the off-board controller 100 or on a server and downloaded to the off-board controller 100. Thus, it should be noted that the off-board controller 100 may in some embodiments be implemented as computer programs stored in memory, e.g. in the computer readable storage unit 502 in Fig. 5, for execution by processors or processing modules, e.g. the processing circuitry 501 in the off-board controller 100 of Fig. 5.
Those skilled in the art will also appreciate that the processing circuitry 501 and the computer readable storage unit 502, i.e. memory, described above may refer to a combination of analog and digital circuits, and/or one or more processors configured with software and/or firmware, e.g. stored in a memory, that when executed by the one or more processors such as the processing circuitry 501 perform as described above. One or more of these processors, as well as the other digital hardware, may be included in a single application-specific integrated circuit (ASIC), or several processors and various digital hardware may be distributed among several separate components, whether individually packaged or assembled into a system-on-a-chip (SoC).
To perform the method actions for remote operation of a working machine 110 comprising a tool 111, the system 10 may comprise an on-board controller 114 having the following arrangement depicted in Fig. 6. Fig. 6 shows a schematic view of an on-board controller 114 according to some embodiments. The on-board controller 114 may comprises a processing circuitry or device 601, a computer readable storage unit 602, a sensor interface 603 and a communication interface 604. The processing circuitry 601 may be arranged to execute instruction sets stored in the computer readable storage unit 602. The on-board controller 114 is configured for arrangement on the working machine 110. In some embodiments, the on-board controller 114 may be configured for controlling the operation of the working machine 110 or to cooperate with other unit or devices (not shown) arranged to control the operation of the working machine 110.
The on-board controller 114 or processing circuitry 601 may further be configured to, or may comprise a receiving module 650 configured to, receive signals from the remote control station 5, e.g. via the communication interface 604, remotely controlling the operation of the working machine 110. Further, the on-board controller 114 or processing circuitry 601 may further be configured to, or may comprise a transmitting module 660 configured to, transmit, e.g. via the communication interface 604, sensor data and camera images to the off-board controller 100. Furthermore, the on-board controller 114 or processing circuitry 601 may further be configured to, or may comprise an obtaining module 670 configured to, obtain sensor data from the at least one sensor 116 on the working machine 110 and camera images from at least one camera 112, 113 on the working machine 110. In some embodiments, the sensor data and camera images may be provided via a communication link 613. The communication link 613 may e.g. be a wired or wireless connection between the sensor interface 603 and at least one sensor 112.
Additionally, the embodiments for remote operation of a working machine 110 comprising a tool 111 described above may be at least partly implemented through one or more processors, such as the processing circuitry 601 in the on-board controller 114 depicted in Fig. 6, together with computer program code for performing the functions and actions of the embodiments herein. The program code mentioned above may also be provided as a computer program product, for instance in the form of a data carrier carrying computer program code or code means for performing the embodiments herein when being loaded into the processing circuitry 601 in the on-board controller 114. The data carrier, or computer readable medium, may be one of an electronic signal, optical signal, radio signal, or computer-readable storage medium. The computer program code may e.g. be provided as pure program code in the on-board controller 114 or on a server and downloaded to the on-board controller 114.Thus, it should be noted that the on-board controller 114 may in some embodiments be implemented as computer programs stored in memory, e.g. in the computer readable storage unit 602 in Fig. 6, for execution by processors or processing modules, e.g. the processing circuitry 601 in the on-board controller 114 of Fig. 6.
Those skilled in the art will also appreciate that the processing circuitry 601 and the computer readable storage unit 602, i.e. memory, described above may refer to a combination of analog and digital circuits, and/or one or more processors configured with software and/or firmware, e.g. stored in a memory, that when executed by the one or more processors such as the processing circuitry 601 perform as described above. One or more of these processors, as well as the other digital hardware, may be included in a single application-specific integrated circuit (ASIC), or several processors and various digital hardware may be distributed among several separate components, whether individually packaged or assembled into a system-on-a-chip (SoC).
The description of the example embodiments provided herein have been presented for purposes of illustration. The description is not intended to be exhaustive or to limit example embodiments to the precise form disclosed, and modifications and variations are possible in light of the above teachings or may be acquired from practice of various alternatives to the provided embodiments. The examples discussed herein were chosen and described in order to explain the principles and the nature of various example embodiments and its practical application to enable one skilled in the art to utilize the example embodiments in various manners and with various modifications as are suited to the particular use contemplated. The features of the embodiments described herein may be combined in all possible combinations of methods, apparatus, modules, systems, and computer program products. It should be appreciated that the example embodiments presented herein may be practiced in any combination with each other.
It should be noted that the word "comprising" does not necessarily exclude the presence of other elements or steps than those listed and the words "a" or "an" preceding an element do not exclude the presence of a plurality of such elements. It should further be noted that any reference signs do not limit the scope of the claims, that the example embodiments may be implemented at least in part by means of both hardware and software, and that several "means", "units" or "devices" may be represented by the same item of hardware.
It should also be noted that the various example embodiments described herein are described in the general context of method steps or processes, which may be implemented in one aspect by a computer program product, embodied in a computer-readable medium, including computer-executable instructions, such as program code, executed by computers in networked environments. A computer-readable medium may include removable and non-removable storage devices including, but not limited to, Read Only Memory (ROM), Random Access Memory (RAM), compact discs (CDs), digital versatile discs (DVD), etc. Generally, program modules may include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. Computer-executable instructions, associated data structures, and program modules represent examples of program code for executing steps of the methods disclosed herein. The particular sequence of such executable instructions or associated data structures represents examples of corresponding acts for implementing the functions described in such steps or processes.
The embodiments herein are not limited to the above described preferred embodiments. Various alternatives, modifications and equivalents may be used. Therefore, the above embodiments should not be construed as limiting.

Claims

A system (10) for remote operation of a working machine (110) comprising a tool (111), characterized in that the system comprises:
a remote control station (5) comprising a work station (102), at least one display (11), and an off-board controller (100);

an on-board controller (114) for arrangement on the working machine and arranged to receive signals from the remote control station (5) remotely controlling the operation of the working machine, wherein the on-board controller is arranged to obtain sensor data from at least one sensor on the working machine and camera images from at least one camera (112, 113) on the working machine;

wherein the off-board controller is arranged to:
receive, via a communication interface (105), sensor data and camera images from the on-board controller (114) on the working machine (110);

analyse the sensor data to determine a location of the tool (111) in relation to surroundings of the working machine (110);

determine at least one overlay (410, 420) indicating the location of the tool (111) in relation to the surroundings of the working machine (110); and

command display of the camera images together with the at least one determined overlay (410, 420) on the at least one display (11) for supporting the operator in determining the location of the tool in relation to the surroundings of the working machine.
The system according to claim 1, wherein the sensor data comprises at least one of a tool height over ground, a tool articulation in relation to a body of the working machine (110), a traveling direction of the working machine (110), and a distance between the tool and a target contact position.
The system according to claim 1, wherein the system further comprises a tool identification sensor (116) for location on the working machine and wherein the off-board controller is arranged to receive information about a tool type from the tool identification sensor (116).
The system according to claim 3, wherein the tool identification sensor (116) comprises a radio frequency identification, RFID, sensor, and wherein the tool comprises an RFID tag comprising information about the tool type.
The system according to any of claims 1-4, wherein the off-board controller is arranged to determine the at least one determined overlay (410, 420) by means of image analysis of the camera images.
The system according to claim 5, wherein the off-board controller is further arranged to provide overlay information related to identification of the tool and tool dimensions in the camera images.
The system according to any of claims 1-6, wherein the at least one determined overlay (410, 420) comprises a virtual shadow on the ground under the tool for indicating the location of the tool.
The system according to any of claims 1-7, wherein the at least one determined overlay (410, 420) comprises an indication of a distance above the ground of the tool or a distance between the tool and a target contact position (150).
The system according to claim 8, wherein the system is arranged to determine the distance above the ground or distance to the target contact position using at least one of LIDAR, radar, ultrasound sensor, angle sensors, inertial measurement unit, or image analysis based on stereo vision images.
The system according to claim 8 or 9, wherein the indication of distance above ground or distance between tool and target contact position comprises a colour indication or a texture indication.
The system according to any of claims 1-10, wherein the at least one determined overlay (410, 420) comprises an interpolated path of travel in relation to a current travel path.
The system according to any of claims 1-11, wherein the off-board controller is further arranged to determine the location of the tool (111) in relation to surroundings of the working machine (110) by determining the location of the tool (111) in relation to a ground below the tool, to a ground adjacent to the working machine 110, and/or to the working machine 110.
A method performed by an off-board controller (100) for remote operation of a working machine (110) comprising a tool (111), the method comprising the steps of:
receiving (201) sensor data and camera images from an on-board controller (114) on the working machine;

analysing (202) the sensor data to determine a location of the tool (111) in relation to the surroundings of the working machine (110);

determining (203) at least one overlay indicating the location of the tool (111) in relation to the surroundings of the working machine; and

commanding (204) display of the camera images together with the determined at least one determined overlay (410, 420) on a display (11) in a control station (5) for supporting the operator in determining the location of the tool in relation to the surroundings of the working machine.
The method according to claim 16, wherein the at least one determined overlay (410, 420) comprises at least one of a virtual shadow on the ground under the tool for indicating the location of the tool, an overlay shape over the tool with a colour, where the colour indicates a range to ground or range to a target contact position, and a future direction of travel of the working machine interpolated from current direction of travel of the working machine.
The method according to claim 16 or 17, wherein the at least one determined overlay (410, 420) comprises a virtual shadow on the ground, and wherein the virtual shadow of the ground is displayed in a colour indicating a range between the tool and ground.