EP4279667A1

EP4279667A1 - Remote operation assistance server and remote operation assistance system

Info

Publication number: EP4279667A1
Application number: EP21931730.2A
Authority: EP
Inventors: Yuki Morita; Masaki Otani; Yusuke Sawada; Yusuke Kamimura; Yoichiro Yamazaki
Original assignee: Kobelco Construction Machinery Co Ltd
Current assignee: Kobelco Construction Machinery Co Ltd
Priority date: 2021-03-19
Filing date: 2021-12-13
Publication date: 2023-11-22
Also published as: CN117043732A; JP2022144601A; WO2022195988A1

Abstract

Provided is a system or the like capable of reducing mental burdens on an operator who remotely operates a work machine, when an image range acquired by a real machine image pickup device loaded on the work machine is changed. A previous notice indicating a future change mode of a real space area projected in a work environment image displayed at a remote image output device 221 is output to a remote output interface 220. It is possible to make the operator who encounters a virtual sound source V<sub>ss</sub> displaced according to a displacement mode of the real space area and/or an icon image A1 in an arrow shape indicating the displacement mode of the real space area recognize beforehand that the real space area projected in the work environment image is to be displaced according to the displacement mode in the future.

Description

Technical Field

The present invention relates to a technology of reporting a state of a work machine to a worker.

Background Art

From a viewpoint of providing a crane which realizes improvement of safety and improvement of work efficiency, a technology of following a subject (a sub hook for example) in an image photographed by a camera, segmenting a part including the subject and displaying the part on a display device has been proposed (for example, see Patent Literature 1).
From a viewpoint of issuing an adequate and appropriate alarm regarding an obstacle existing around a work machine, a technology of calculating a dead angle area from a driver's seat of a work machine (hydraulic shovel) based on information of a posture and a movement of the work machine, determining a visual recognition level for an obstacle based on the dead angle area and a positional relation between the obstacle and the work machine and correcting an alarm level according to the visual recognition level has been proposed (for example, see Patent Literature 2).

Citation List

Patent Literature

Patent Literature 1: Japanese Patent Laid-Open No. 2019-156533
Patent Literature 2: Japanese Patent No. 6581139

Summary of Invention

Technical Problem

However, in a case where a work machine is remotely operated, when an image range which is acquired by a real machine image pickup device loaded on the work machine and provided for an operator is rapidly changed in order to follow an object or the like, there is a possibility that the operator is made to have increased mental burdens such as anxiety.
Therefore, it is an object of the present invention to provide a system or the like capable of reducing mental burdens on an operator who remotely operates a work machine, when an image range acquired by a real machine image pickup device loaded on the work machine is changed.

Solution to Problem

A remote operation assistance server of the present invention is
a server for assisting a remote operation of a work machine using a remote operation device, and comprises:

a first assistance processing element configured to acquire a picked-up image indicating an environment of the work machine through a real machine image pickup device loaded on the work machine, and make a work environment image according to the picked-up image be displayed at a remote image output device configuring the remote operation device; and
a second assistance processing element configured to recognize a future change mode of a real space area projected in the work environment image displayed at the remote image output device by the first assistance processing element, and make a remote output interface configuring the remote operation device output a previous notice indicating the future change mode of the real space area.

According to the remote operation assistance server of the configuration, the previous notice indicating the future change mode of the real space area projected in the work environment image displayed at the remote image output device is output by the remote output interface configuring the remote operation device. For example, a space occupancy mode of the real space area projected in the work environment image may be changed according to switching of one real machine image pickup device corresponding to the work environment image among the plurality of real machine image pickup devices and image processing by the first assistance processing element or the like in addition to an optical axis direction and/or a photographing magnification of the real machine image pickup device. Then, it is possible to make an operator who receives the previous notice recognize the future change mode of the real space area projected in the work environment image beforehand. Therefore, mental burdens on the operator when the space occupancy mode of the real space area projected in the work environment image displayed at the remote image output device is changed according to the mode as the previous notice immediately thereafter are reduced.

Brief Description of Drawings

FIG. 1 is an explanatory drawing regarding a configuration of a remote operation assistance system.
FIG. 2 is an explanatory drawing regarding a configuration of a remote operation device.
FIG. 3 is an explanatory drawing regarding a configuration of a work machine.
FIG. 4 is an explanatory drawing regarding a function of the remote operation assistance system of a first embodiment.
FIG. 5 is an explanatory drawing regarding a work environment image.
FIG. 6 is an explanatory drawing regarding a first displacement mode of a virtual sound source.
FIG. 7 is an explanatory drawing regarding a second displacement mode of the virtual sound source.
FIG. 8 is an explanatory drawing regarding a function of a remote operation assistance system of a second embodiment.

Description of Embodiments

(Configuration of remote operation assistance system)

A remote operation assistance system illustrated in FIG. 1 is configured by a remote operation assistance server 10, a remote operation device 20, and a work machine 40 which can be remotely operated via the remote operation device 20. Only one of the remote operation device 20 and the work machine 40 may be a component of the remote operation assistance system. The remote operation assistance server 10, the remote operation device 20 and the work machine 40 are configured to perform network communication with each other. An intercommunication network of the remote operation assistance server 10 and the remote operation device 20 and an intercommunication network of the remote operation assistance server 10 and the work machine 40 may be the same or may be different.

(Configuration of remote operation assistance server)

The remote operation assistance server 10 comprises a database 102, a first assistance processing element 121, and a second assistance processing element 122. The database 102 stores and holds picked-up image data or the like. The database 102 may be configured by a database server different from the remote operation assistance server 10. Each assistance processing element is configured by an arithmetic processing unit (a single-core processor, a multi-core processor, or a processor core configuring it), reads required data and software from a storage device such as a memory and executes arithmetic processing to be described later according to the software with the data as a target.

(Configuration of remote operation device)

The remote operation device 20 comprises a remote controller 200, a remote input interface 210, and a remote output interface 220. The remote controller 200 is configured by an arithmetic processing unit (a single-core processor, a multi-core processor, or a processor core configuring it), reads required data and software from a storage device such as a memory and executes arithmetic processing according to the software with the data as a target.
The remote input interface 210 comprises a remote operation mechanism 211. The remote output interface 220 comprises a remote image output device 221, a remote sound output device 222, and a remote radio communication device 224.
The remote operation mechanism 211 includes a traveling operation device, a turning operation device, a boom operation device, an arm operation device, and a bucket operation device. Each operation device comprises an operation lever which receives a rotating operation. The operation lever (traveling lever) of the traveling operation device is operated so as to move a lower traveling body 410 of the work machine 40. The traveling lever may also serve as a traveling pedal. For example, the traveling pedal fixed to a base part or a lower end part of the traveling lever may be provided. The operation lever (turning lever) of the turning operation device is operated so as to move a hydraulic type turning motor configuring a turning mechanism 430 of the work machine 40. The operation lever (boom lever) of the boom operation device is operated so as to move a boom cylinder 442 of the work machine 40. The operation lever (arm lever) of the arm operation device is operated so as to move an arm cylinder 444 of the work machine 40. The operation lever (bucket lever) of the bucket operation device is operated so as to move a bucket cylinder 446 of the work machine 40.
Each operation lever configuring the remote operation mechanism 211 is, for example, as illustrated in FIG. 2, arranged around a seat St for an operator to sit. The seat St is in a form like a high back chair with armrests, but may be a sitting part in an optional form that the operator can sit, such as a form like a low back chair without a headrest or a form like a chair without a backrest.
A pair of left and right traveling levers 2110 according to left and right crawlers are arranged laterally side by side at a front of the seat St. One operation lever may serve as a plurality of operation levers. For example, a left side operation lever 2111 provided at the front of a left side frame of the seat St illustrated in FIG. 2 may function as the arm lever when operated in front and back directions and may function as the turning lever when operated in left and right directions. Similarly, a right side operation lever 2112 provided at the front of a right side frame of the seat St illustrated in FIG. 2 may function as the boom lever when operated in the front and back directions and may function as the bucket lever when operated in the left and right directions. A lever pattern may be arbitrarily changed by an operation instruction of the operator.
The remote image output device 221 is configured by a center remote image output device 2210, a left side remote image output device 2211, and a right side remote image output device 2212 having a screen in a roughly rectangular shape arranged respectively at the front, diagonally left front, and diagonally right front of the seat St, as illustrated in FIG. 2 for example. Shapes and sizes of the respective screens (image display areas) of the center remote image output device 2210, the left side remote image output device 2211, and the right side remote image output device 2212 may be the same or may be different.
As illustrated in FIG. 2, a right edge of the left side remote image output device 2211 is adj acent to a left edge of the center remote image output device 2210 so that the screen of the center remote image output device 2210 and the screen of the left side remote image output device 2211 form an inclination angle θ1 (for example, 120° ≤ θ1 ≤ 150°). As illustrated in FIG. 2, a left edge of the right side remote image output device 2212 is adj acent to a right edge of the center remote image output device 2210 so that the screen of the center remote image output device 2210 and the screen of the right side remote image output device 2212 form an inclination angle θ2 (for example, 120° ≤ θ2 ≤ 150°). The inclination angles θ1 and θ2 may be the same or may be different.
The respective screens of the center remote image output device 2210, the left side remote image output device 2211, and the right side remote image output device 2212 may be parallel to a vertical direction or may be inclined relative to the vertical direction. At least one remote image output device of the center remote image output device 2210, the left side remote image output device 2211, and the right side remote image output device 2212 may be configured by multi-divisional remote image output devices. For example, the center remote image output device 2210 may be configured by a pair of vertically adjacent remote image output devices having the screen in the roughly rectangular shape.
The remote sound output device 222 is configured by a center sound output device 2220, a left side sound output device 2221, and a right side sound output device 2222 configured by one or more speakers and arranged respectively at the back of the seat St, a left armrest rear part, and a right armrest rear part, as illustrated in FIG. 2 for example. Respective specifications of the center sound output device 2220, the left side sound output device 2221, and the right side sound output device 2222 may be the same or may be different. The remote sound output device 222 realizes a virtual sound source V_ss which is sound capable of position displacement around the seat St.

(Configuration of work machine)

The work machine 40 comprises a real machine controller 400, a real machine input interface 41, a real machine output interface 42, and a work mechanism 440. The real machine controller 400 comprises an image processor 30. The image processor 30 comprises a state detection element 31, an image prediction element 32 and an image compression element 34. Each of the components of the real machine controller 400 and the image processor 30 is configured by an arithmetic processing unit (a single-core processor, a multi-core processor or a processor core configuring it), reads required data and software from a storage device such as a memory and executes arithmetic processing according to the software with the data as a target.
The work machine 40 is a crawler shovel (construction machine) for example, and comprises, as illustrated in FIG. 3, a crawler type lower traveling body 410 and an upper turning body 420 loaded on the lower traveling body 410 so as to be turnable through the turning mechanism 430. A cab 424 (driving room) is provided on a front left side part of the upper turning body 420. The work mechanism 440 is provided on a front center part of the upper turning body 420.
The real machine input interface 41 comprises a real machine operation mechanism 411, a real machine image pickup device 412, and a real machine periphery monitoring device 414. The real machine operation mechanism 411 comprises a plurality of operation levers arranged similarly to the remote operation mechanism 211 around the seat arranged inside the cab 424. A driving mechanism or a robot which receives a signal according to an operation mode of a remote operation lever and moves a real machine operation lever based on the received signal is provided in the cab 424. The real machine image pickup device 412 is installed inside the cab 422 for example, and picks up an image of an environment including at least a part of the work mechanism 440 over a front window and a pair of left and right side windows. A part or all of the front window and the side windows may be omitted. The real machine periphery monitoring device 414 is configured by a ranging sensor or the like for detecting presence of a target object (for example, a worker, another work machine, and/or a vehicle) in a real space area (for example, a right side area, a left side area and/or a rear area and/or a dead angle area of the cab 424) where the image cannot be picked up by the real machine image pickup device 412 in a basic posture (for example, the posture in which an optical axis is turned to the front).
The real machine output interface 42 comprises a real machine radio communication device 422.
As illustrated in FIG. 3, the work mechanism 440 as a work mechanism comprises a boom 441 mounted on the upper turning body 420 so as to be hoistable, an arm 443 rotatably connected to a distal end of the boom 441, and a bucket 445 rotatably connected to a distal end of the arm 443. On the work mechanism 440, the boom cylinder 442, the arm cylinder 444, and the bucket cylinder 446 configured by extendable hydraulic cylinders are mounted.
The boom cylinder 442 is interposed between the boom 441 and the upper turning body 420 so as to be extended and contracted by receiving supply of hydraulic oil and rotate the boom 441 in a hoisting direction. The arm cylinder 444 is interposed between the arm 443 and the boom 441 so as to be extended and contracted by receiving the supply of the hydraulic oil and rotate the arm 443 around a horizonal axis to the boom 441. The bucket cylinder 446 is interposed between the bucket 445 and the arm 443 so as to be extended and contracted by receiving the supply of the hydraulic oil and rotate the bucket 445 around the horizontal axis to the arm 443.

(Function)

FIG. 4 is a flowchart explaining the function of the remote operation assistance system of the configuration described above. In the flowchart, a block "C•" is used to simplify description, means transmission and/or reception of data and means a conditional branch of executing processing in a branch direction on condition that the data is transmitted and/or received.
In the remote operation device 20, an environment checking request is transmitted to the remote operation assistance server 10 via the remote radio communication device 224 (FIG. 4/STEP 210). For example, a specifying operation performed via the remote input interface 210 by an operator may be determined as a transmission start requirement for the environment checking request. The "specifying operation" is an operation of tapping or the like in the remote input interface 210 for specifying the work machine 40 that the operator intends to remotely control, for example.
In the remote operation assistance server 10, when the environment checking request is received, the environment checking request is transmitted to the pertinent work machine 40 by the first assistance processing element 121 (FIG. 4/C10).
In the work machine 40, when the environment checking request is received via the real machine radio communication device 422 (FIG. 4/C40), a picked-up image is acquired via the real machine image pickup device 412 by the real machine controller 400, and picked-up image data indicating the picked-up image is transmitted to the remote operation assistance server 10 via the real machine radio communication device 422 (FIG. 4/STEP 410).
In the remote operation assistance server 10, when the picked-up image data is received by the first assistance processing element 121 (FIG. 4/C11), work environment image data according to the picked-up image is transmitted to the remote operation device 20 by the second assistance processing element 122 (FIG. 4/STEP 110). The work environment image data is, in addition to the picked-up image data itself, image data indicating a simulated work environment image generated based on the picked-up image.
In the remote operation device 20, when the work environment image data is received via the remote radio communication device 224 (FIG. 4/C21), a work environment image according to the work environment image data is output to the remote image output device 221 by the remote controller 200 (FIG. 4/STEP 212).
Thus, for example, as illustrated in FIG. 5, the work environment image projecting the boom 441 and the arm 443 as a part of the work mechanism 440 and a heap of rubble or soil (which is a work target by the bucket 445) in front of the cab 424 through a window frame demarcating the cab 424 is output to the remote image output device 221 (in particular, the center remote image output device 2210).
In the work machine 40, by the real machine controller 400, whether or not the work machine 40 is in a specified state is determined (FIG. 4/STEP 411). At the time, which specified state of the plurality of predetermined different specified states the specified state is pertinent to may be determined. The "specified state" is a state where the real machine controller 400 is set or programmed beforehand so as to make the real machine image pickup device 412 demonstrate the pan function and/or tilt function and the zoom function as the work machine 40 is turned to the specified state.
For example, a state where the presence of the target object is detected in the dead angle area of the real machine image pickup device 412 in the normal posture by the real machine periphery monitoring device 414 may be defined as the specified state. The left side area of the cab 424 is the dead angle area of the real machine image pickup device 412 in the normal posture, and when the presence of the target object is detected in the left side area, the optical axis is turned to a left direction by the pan function of the real machine image pickup device 412 and the optical axis is turned downward (or upward) by the tilt function as needed. Similarly, the right side area of the cab 424 is the dead angle area of the real machine image pickup device 412 in the normal posture, and when the presence of the target object is detected in the right side area, the optical axis is turned to a right direction by the pan function of the real machine image pickup device 412 and the optical axis is turned downward (or upward) by the tilt function as needed.
For example, a state where the presence of the target object of a specified kind is detected in the real space area to be an image pickup target of the real machine image pickup device 412 may be defined as the specified state. In the case where the target object (for example, a human such as a worker) of a first specified kind is detected, by a zoom-in function of the real machine image pickup device 412, a display magnification of an image area including the target object is increased. In the case where the presence of the target object (for example, another work machine) of a second specified kind is detected, by a zoom-out function of the real machine image pickup device 412, the display magnification of the image area including the target object is reduced. Together with the zoom function of the real machine image pickup device 412, the pan function and/or the tilt function may be controlled in order to adjust a position of the target object in the picked-up image.
For example, a state where it is detected that a displacement amount and/or a displacement speed of the target object is a predetermined value or greater in the real space area to be the image pickup target of the real machine image pickup device 412 may be defined as the specified state. In the case where the displacement amount of the target object (for example, the bucket 445) exceeds the predetermined value, by the pan function and/or the tilt function of the real machine image pickup device 412, the direction of the optical axis of the real machine image pickup device 412 is changed so as to follow the target object.
When it is determined that the work machine 40 is in the specified state (FIG. 4/STEP 411..YES), by the real machine controller 400, a control mode of a posture change function (the pan function and/or the tilt function) and/or the zoom function of the real machine image pickup device 412 according to the specified state is recognized, and the recognition result is transmitted to the remote operation assistance server 10 via the real machine radio communication device 422 (FIG. 4/STEP 412). A correspondence relation between (the kind of) the specified state and the control mode of the posture change function and/or the zoom function of the real machine image pickup device 412 may be stored in a storage device configuring the real machine controller 400 or the database 102.
When it is determined that the work machine 40 is not in the specified state (FIG. 4/STEP 411..NO), subsequent processing is executed without recognizing the control mode of the zoom function or the like of the real machine image pickup device 412.
In the remote operation assistance server 10, when the control mode of the posture change function and/or the zoom function of the real machine image pickup device 412 is received (FIG. 4/C 12), by the second assistance processing element 122, a change mode of the real space area projected in the picked-up image (consequently the work environment image) is recognized and then transmitted to the work machine 40 (FIG. 4/STEP 112). In the case where the direction of the optical axis is changed to the left and right by the pan function of the real machine image pickup device 412, by the second assistance processing element 122, it is recognized that the real space area projected in the picked-up image is similarly displaced to the left and right. In the case where the direction of the optical axis is changed up and down by the tilt function of the real machine image pickup device 412, by the second assistance processing element 122, it is recognized that the real space area projected in the picked-up image is similarly displaced up and down. In the case where a zoom magnification is increased by the zoom-in function of the real machine image pickup device 412, by the second assistance processing element 122, it is recognized that the real space area projected in the picked-up image is enlarged. In the case where the zoom magnification is reduced by the zoom-out function of the real machine image pickup device 412, by the second assistance processing element 122, it is recognized that the real space area projected in the picked-up image is reduced.
In the remote operation device 20, when the change mode of the real space area is received via the remote radio communication device 224 (FIG. 4/C22), by the remote controller 200, a previous notice indicating the change mode is output via the remote output interface 220 (FIG. 4/STEP 214).
Thus, for example, when it is recognized as the change mode of the real space area that the real space area projected in the picked-up image is to be displaced in the left direction, as illustrated in FIG. 6, the virtual sound source V_ss realized by the remote sound output device 222 is controlled so as to be displaced in the left direction (the same direction as the displacement direction of the real space area). In this case, in addition to or instead of the displacement of the virtual sound source V_ss, as illustrated in FIG. 6, a graphic or icon A1 in an arrow shape in the left direction (the same direction as the displacement direction of the real space area) may be output to the remote image output device 221. In addition, a specified icon may be output so as to be moved in the same direction as the virtual sound source V_ss in the remote image output device 221.
In addition, for example, when it is recognized as the change mode of the real space area that the real space area projected in the picked-up image is to be reduced, as illustrated in FIG. 7, control is performed so that the sound gradually becomes strong and/or a frequency of the sound gradually becomes high in a state where the virtual sound source V_ss realized by the remote sound output device 222 is localized. In this case, in addition to or instead of strength of the virtual sound source V_ss and/or a height of the frequency, as illustrated in FIG. 7, a graphic or icon A2 in the arrow shape toward an image area S enlarged by zoom-in and/or an outer side of the image area S may be output to the remote image output device 221. Further, a specified icon may be output so as to be scaled according to scaling of the real space area or the image area in the remote image output device 221.
On the other hand, when it is recognized as the change mode of the real space area that the real space area projected in the picked-up image is to be enlarged, the control is performed so that the sound gradually becomes weak and/or the frequency of the sound gradually becomes low in the state where the virtual sound source V_ss realized by the remote sound output device 222 is localized. In this case, in addition to or instead of the strength of the virtual sound source V_ss and/or the height of the frequency, the graphic or icon A2 in the arrow shape toward the image area reduced by zoom-out and/or an inner side of the image area may be output to the remote image output device 221.
Thereafter, by the remote controller 200, a previous notice output completion report indicating that the previous notice has been output via the remote output interface 220 is transmitted to the remote operation assistance server 10 via the remote radio communication device 224 (FIG. 4/STEP 216).
In the remote operation assistance server 10, when the previous notice output completion report is received, the previous notice output completion report is transmitted to the work machine 40 by the second assistance processing element 122 (FIG. 4/C 14).
In the work machine 40, when the previous notice output completion report is received via the real machine radio communication device 422 (FIG. 4/C44), the posture change function and/or the zoom function of the real machine image pickup device 412 is controlled in a form according to the specified state by the real machine controller 400 (FIG. 4/STEP 414).
Thus, the real space area projected in the picked-up image acquired via the real machine image pickup device 412, consequently the work environment image output at the remote image output device 221, is changed (see FIG. 4/STEP 410 → .. → STEP 212, FIG. 5).
For example, when the optical axis direction is changed from the front of the work machine 40 to the diagonally left front by the control of the pan function of the real machine image pickup device 412, the real space area projected in the work environment image is changed from a front area of the work machine 40 to a diagonally left front area. In addition, when the optical axis direction is changed from a front horizontal direction to a diagonally lower front by the control of the tilt function of the real machine image pickup device 412, the real space area projected in the work environment image is changed to an area lower than before.
For example, by controlling the zoom-in function of the real machine image pickup device 412 and reducing the zoom magnification, the real space area projected in the work environment image is reduced (or the image area is enlarged). In addition, by controlling the zoom-out function of the real machine image pickup device 412 and increasing the zoom magnification, the real space area projected in the work environment image is enlarged (or the image area is reduced).
In the remote operation device 20, an operation mode of the remote operation mechanism 211 is recognized by the remote controller 200, and a remote operation command according to the operation mode is transmitted to the remote operation assistance server 10 via the remote radio communication device 224 (FIG. 4/STEP 220).
In the remote operation assistance server 10, when the remote operation command is received by the second assistance processing element 122, the remote operation command is transmitted to the work machine 40 by the first assistance processing element 121 (FIG. 4/C16).
In the work machine 40, when the operation command is received via the real machine radio communication device 422 by the real machine controller 400 (FIG. 4/C46), operations of the work mechanism 440 or the like are controlled (FIG. 4/STEP 420). For example, work of scooping soil in front of the work machine 40 by the bucket 445, turning the upper turning body 420 and then dropping the soil from the bucket 445 is executed.

(Effects)

According to the remote operation assistance system of the configuration, the previous notice indicating a future change mode of the rear space area projected in the work environment image displayed at the remote image output device 221 is output to the remote output interface 220.
Specifically, it is possible to make an operator who encounters the virtual sound source V_ss displaced according to the displacement mode of the real space area and/or the icon image A1 in the arrow shape indicating the displacement mode of the real space area recognize beforehand that the real space area projected in the work environment image is to be displaced according to the displacement mode in the future (see FIG. 6). For example, a space occupancy mode of the real space area projected in the work environment image may be displaced according to the change of the optical axial direction of the real machine image pickup device 412 or the like (see FIG. 5). Therefore, mental burdens on the operator when the real space area projected in the work environment image displayed at the remote image output device 221 is displaced as the previous notice are reduced.
Further, it is possible to make the operator who encounters the virtual sound source V_ss which emits the sound for which the strength and/or the height of the frequency is changed according to a scaling mode of the real space area and/or the icon A2 in the arrow shape indicating the scaling mode of the real space area recognize beforehand that the real space area projected in the work environment image is to be scaled according to the scaling mode in the future (see FIG. 7). For example, the space occupancy mode of the real space area projected in the work environment image may be enlarged or reduced according to the change of a photographing magnification of the real machine image pickup device 412. Therefore, the mental burdens on the operator when the real space area projected in the work environment image displayed at the remote image output device 221 is enlarged or reduced as the previous notice are reduced.

(Other embodiments of present invention)

In the embodiment described above, the first assistance processing element 121 and the second assistance processing element 122 are configured by the remote operation assistance server 10, however, as another embodiment, the first assistance processing element 121 and/or the second assistance processing element 122 may be configured by the work machine 40 and/or the remote operation device 20.
In the embodiment described above, the control mode of the posture change function and/or the zoom function of the real machine image pickup device 412 according to the specified state is recognized by the real machine controller 400 as it is recognized that the work machine 40 is in the specified state (see FIG. 4/STEP 411..YES → STEP 412), however, as another embodiment, the control mode of the posture change function and/or the zoom function of the real machine image pickup device 412 according to the specified state may be recognized by the remote operation assistance server 10 or the second assistance processing element 122.
In this case, a series of processing may be executed according to a flowchart illustrated in FIG. 8 instead of the flowchart illustrated in FIG. 4. The block "C•" is used to simplify the description, means the transmission and/or the reception of the data and means the conditional branch of executing the processing in the branch direction on condition that the data is transmitted and/or received. In FIG. 8, for the processing in common with FIG. 4, a same sign is used and the explanation is omitted.
When it is determined that the work machine 40 is in the specified state (FIG. 8/STEP 411..YES), by the real machine controller 400, the specified state is transmitted to the remote operation assistance server 10 via the real machine radio communication device 422 (FIG. 8/STEP 413).
In the remote operation assistance server 10, when the specified state is received (FIG. 8/C13), by the second assistance processing element 122, the control mode of the posture change function (the pan function and/or the tilt function) and/or the zoom function of the real machine image pickup device 412 according to the specified state is recognized (FIG. 8/STEP 111). Further, by the second assistance processing element 122, the change mode of the real space area projected in the picked-up image (consequently the work environment image) according to the control mode of the posture change function and/or the zoom function of the real machine image pickup device 412 is recognized and then transmitted to the work machine 40 (FIG. 8/STEP 112).
Thereafter, in the remote operation assistance server 10, when the previous notice output completion report is received from the remote operation device 20 (FIG. 8/C 15), by the second assistance processing element 122, a control command for the posture change function and/or the zoom function of the real machine image pickup device 412 is generated in the form according to the specified state, and is transmitted to the work machine 40 (FIG. 8/STEP 114).
In the work machine 40, when the control command is received via the real machine radio communication device 422 (FIG. 8/C44), by the real machine controller 400, the posture change function and/or the zoom function of the real machine image pickup device 412 is controlled according to the control command (FIG. 8/STEP 414).
Thus, the real space area projected in the picked-up image acquired via the real machine image pickup device 412, consequently the work environment image output at the remote image output device 221, is changed (see FIG. 4/STEP 410 → .. →STEP 212, FIG. 5).
In the embodiment described above, the control mode of the posture change function and/or the zoom function of the real machine image pickup device 412 according to the specified state is recognized by the real machine controller 400 as it is recognized that the work machine 40 is in the specified state (see FIG. 4/STEP 411..YES → STEP 412), however, as another embodiment, the control mode of the posture change function and/or the zoom function of the real machine image pickup device 412 according to the specified state may be recognized by the remote operation device 20 or the remote controller 200.
In the embodiment described above, the real space area projected in the picked-up image and the corresponding work environment image is continuously displaced by the pan function and/or the tilt function of the real machine image pickup device 412, however, as another embodiment, the plurality of real machine image pickup devices having different image pickup areas may be loaded on the work machine 40 and the real space area projected in the work environment image may be discontinuously displaced by switching the real machine image pickup device which acquires the picked-up image to be a base of the work environment image from one real machine image pickup device to another real machine image pickup device. For example, one real machine image pickup device may be switched to another real machine image pickup device the image pickup area of which is the real space area where the target object is detected by the real machine periphery monitoring device 414. Even in this case, by the previous notice output by the remote output interface 220, the mental burdens put on the operator are reduced even when the work environment image output at the remote image output device 221 is discontinuously displaced or changed.
In the embodiment described above, the real space area projected in the picked-up image and the corresponding work environment image is continuously displaced by the pan function and/or the tilt function of the real machine image pickup device 412, however, as another embodiment, the real space area projected in the work environment image may be displaced and/or scaled by image processing when the work environment image is generated by the first assistance processing element 121. Even in this case, by the previous notice output by the remote output interface 220, the mental burdens put on the operator are reduced even when the work environment image output at the remote image output device 221 is displaced and/or scaled.
In the embodiment described above, the zoom-in and the zoom-out of the real machine image pickup device 412 are previously noticed by the strength (the height of a sound pressure) of the sound emitted from the virtual sound source V_ss and/or the height of the frequency (see FIG. 7). As another embodiment, the real machine image pickup device 412 may have only the pan function, and the zoom-in and the zoom-out of the real machine image pickup device 412 may be previously noticed by the up and down displacement of the virtual sound source V_ss before and after the change of a pan angle is previously noticed by the left and right displacement of the virtual sound source V_ss (see FIG. 6). Similarly, as another embodiment, the real machine image pickup device 412 may have only the tilt function, and the zoom-in and the zoom-out of the real machine image pickup device 412 may be previously noticed by the left and right displacement of the virtual sound source V_ss before and after the change of a tilt angle is previously noticed by the up and down displacement of the virtual sound source V_ss.
In the remote operation assistance server of the configuration described above, it is preferable that the second assistance processing element recognizes the future change mode of the real space area projected in the work environment image, according to the specified state detected by the real machine periphery monitoring device loaded on the work machine.
According to the remote operation assistance server of the configuration, it is possible to make the operator who receives the previous notice of the form of the displacement of an output part recognize beforehand that the real space area projected in the work environment image is to be displaced according to the displacement mode in the future and also the work machine is in the specified state. Therefore, the mental burdens on the operator when the real space area projected in the work environment image displayed at the remote image output device is displaced as the previous notice are reduced.
In the remote operation assistance server of the configuration described above, it is preferable that the second assistance processing element recognizes the future displacement mode of the real space area as the future change mode of the real space area, and makes the output part of the previous notice configured by the remote output interface be displaced according to the future displacement mode of the real space area.
According to the remote operation assistance server of the configuration, it is possible to make the operator who receives the previous notice of the form of the displacement of the output part recognize beforehand that the real space area projected in the work environment image is to be displaced according to the displacement mode in the future. For example, the space occupancy mode of the real space area projected in the work environment image may be displaced according to the change of the optical axial direction of the real machine image pickup device, switching of one real machine image pickup device corresponding to the work environment image among the plurality of real machine image pickup devices and the image processing by the first assistance processing element or the like. Therefore, the mental burdens on the operator when the real space area projected in the work environment image displayed at the remote image output device is displaced as the previous notice are reduced.
In the remote operation assistance server of the configuration described above, it is preferable that the second assistance processing element further recognizes a future scaling mode of the real space area as the future change mode of the real space area, and makes the remote output interface displace the output part of the previous notice according to the future scaling mode of the real space area before or after making the output part of the previous notice configured by the remote output interface be displaced according to the future displacement mode of the real space area.
According to the remote operation assistance server of the configuration, it is possible to make the operator who receives the previous notice of the form of the displacement of the output part over two stages recognize beforehand that the real space area projected in the work environment image is to be displaced according to a first displacement mode and is to be scaled according to a second displacement mode in the future. Therefore, the mental burdens on the operator when the real space area projected in the work environment image displayed at the remote image output device is displaced and scaled as the previous notice are reduced.
In the remote operation assistance server of the configuration described above, it is preferable that the second assistance processing element makes the remote sound output device configuring the remote output interface displace the virtual sound source that emits the sound as the previous notice.
According to the remote operation assistance server of the configuration, it is possible to make the operator who receives the previous notice of the form of the displacement of the virtual sound source recognize beforehand that the real space area projected in the work environment image is to be displaced (and is to be scaled further) according to the displacement mode in the future. For example, when the virtual sound source is displaced to right (left, up and down), it is possible to make the operator recognize that the real space area projected in the work environment image is to be displaced to right (left, up and down) immediately after. At the time, since the previous notice is reported to the operator not through visual sensation but through auditory sensation, a situation of distracting a visual attention of the operator to the work environment image is avoided. Therefore, the mental burdens on the operator when the real space area projected in the work environment image displayed at the remote image output device is displaced as the previous notice are reduced.
In the remote operation assistance server of the configuration described above, it is preferable that the second assistance processing element recognizes the future displacement mode of the real space area as the future change mode of the real space area according to a switching mode from one real machine image pickup device of the plurality of real machine image pickup devices loaded on the work machine to another real machine image pickup device.
According to the remote operation assistance server of the configuration, it is possible to make the operator who receives the previous notice of the change of the output mode recognize beforehand that the future scaling mode of the real space area projected in the work environment image is to be changed according to the change of the output mode resulting from switching of the real machine image pickup device. Therefore, the mental burdens on the operator when the real space area projected in the work environment image displayed at the remote image output device is enlarged or reduced as the previous notice are reduced.
In the remote operation assistance server of the configuration described above, it is preferable that the second assistance processing element recognizes the future scaling mode of the real space area as the future change mode of the real space area, and makes the remote output interface change the output mode of the previous notice according to the future scaling mode of the real space area.
According to the remote operation assistance server of the configuration, it is possible to make the operator who receives the previous notice of the change of the output mode recognize beforehand that the future scaling mode of the real space area projected in the work environment image is to be changed according to the change of the output mode. For example, the space occupancy mode of the real space area projected in the work environment image may be enlarged or reduced according to the photographing magnification of the real machine image pickup device, the switching of one real machine image pickup device corresponding to the work environment image among the plurality of real machine image pickup devices and the image processing by the first assistance processing element or the like. Therefore, the mental burdens on the operator when the real space area projected in the work environment image displayed at the remote image output device is enlarged or reduced as the previous notice are reduced.
In the remote operation assistance server of the configuration described above, it is preferable that the second assistance processing element makes the remote sound output device configuring the remote output interface change at least one of the height of a sound pressure and the height of the frequency of the sound as the previous notice.
According to the remote operation assistance server of the configuration, it is possible to make the operator who receives the previous notice as the sound for which the height of the sound pressure and/or the height of the frequency is changed recognize beforehand that the real space area projected in the work environment image is to be scaled according to the change mode in the future. For example, when the sound pressure gradually becomes high, it is possible to make the operator recognize beforehand that the real space area is to be reduced. On the other hand, when the sound pressure gradually becomes low, it is possible to make the operator recognize beforehand that the real space area is to be enlarged. At the time, since the previous notice is reported to the operator not through the visual sensation but through the auditory sensation, the situation of distracting the visual attention of the operator to the work environment image is avoided. Therefore, the mental burdens on the operator when the real space area projected in the work environment image displayed at the remote image output device is enlarged or reduced as the previous notice are reduced.

Reference Signs List

10..remote operation assistance server, 20..remote operation device, 40..work machine, 41..real machine input interface, 42..real machine output interface, 102..database, 121..first assistance processing element, 122..second assistance processing element, 200..remote controller, 210..remote input interface, 211..remote operation mechanism, 220..remote output interface, 221..remote image output device, 222..remote sound output device, 400..real machine controller, 410..lower traveling body, 420..upper turning body, 424.. cab (driving room), 440..work mechanism, 445..bucket (work part).

Claims

A remote operation assistance server, which is a server for assisting a remote operation of a work machine using a remote operation device, comprising:
a first assistance processing element configured to acquire a picked-up image indicating an environment of the work machine through a real machine image pickup device loaded on the work machine, and make a work environment image according to the picked-up image be displayed at a remote image output device configuring the remote operation device; and

a second assistance processing element configured to recognize a future change mode of a real space area projected in the work environment image displayed at the remote image output device by the first assistance processing element, and make a remote output interface configuring the remote operation device output a previous notice indicating the future change mode of the real space area.
The remote operation assistance server according to claim 1,
wherein the second assistance processing element recognizes the future change mode of the real space area projected in the work environment image, according to a specified state detected by a real machine periphery monitoring device loaded on the work machine.
The remote operation assistance server according to claim 1 or 2,
wherein the second assistance processing element recognizes a future displacement mode of the real space area as the future change mode of the real space area, and makes an output part of the previous notice configured by the remote output interface be displaced according to the future displacement mode of the real space area.
The remote operation assistance server according to claim 3,
wherein the second assistance processing element further recognizes a future scaling mode of the real space area as the future change mode of the real space area, and makes the remote output interface displace the output part of the previous notice according to the future scaling mode of the real space area before or after making the output part of the previous notice configured by the remote output interface be displaced according to the future displacement mode of the real space area.
The remote operation assistance server according to claim 3 or 4,
wherein the second assistance processing element makes a remote sound output device configuring the remote output interface displace a virtual sound source that emits sound as the previous notice.
The remote operation assistance server according to any one of claims 3 to 5,
wherein the second assistance processing element recognizes the future displacement mode of the real space area as the future change mode of the real space area according to a switching mode from one real machine image pickup device of the plurality of real machine image pickup devices loaded on the work machine to another real machine image pickup device.
The remote operation assistance server according to any one of claims 1 to 6,
wherein the second assistance processing element recognizes a future scaling mode of the real space area as the future change mode of the real space area, and makes the remote output interface change an output mode of the previous notice according to the future scaling mode of the real space area.
The remote operation assistance server according to any one of claims 1 to 7,
wherein the second assistance processing element makes a remote sound output device configuring the remote output interface change at least one of a height of a sound pressure and a height of a frequency of sound as the previous notice.
A remote operation assistance system configured by the remote operation assistance server according to any one of claims 1 to 8 and at least one of the remote operation device and the work machine.