US20240141616A1 - Remote operation assistance server and remote operation asistance system - Google Patents

Remote operation assistance server and remote operation asistance system Download PDF

Info

Publication number
US20240141616A1
US20240141616A1 US18/278,209 US202118278209A US2024141616A1 US 20240141616 A1 US20240141616 A1 US 20240141616A1 US 202118278209 A US202118278209 A US 202118278209A US 2024141616 A1 US2024141616 A1 US 2024141616A1
Authority
US
United States
Prior art keywords
remote
space area
image
real space
remote operation
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
US18/278,209
Other languages
English (en)
Inventor
Yuki Morita
Masaki Otani
Yusuke SAWADA
Yusuke Kamimura
Yoichiro Yamazaki
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Kobelco Construction Machinery Co Ltd
Original Assignee
Kobelco Construction Machinery Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Kobelco Construction Machinery Co Ltd filed Critical Kobelco Construction Machinery Co Ltd
Assigned to KOBELCO CONSTRUCTION MACHINERY CO., LTD. reassignment KOBELCO CONSTRUCTION MACHINERY CO., LTD. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: KAMIMURA, Yusuke, MORITA, YUKI, OTANI, MASAKI, SAWADA, YUSUKE, YAMAZAKI, YOICHIRO
Publication of US20240141616A1 publication Critical patent/US20240141616A1/en
Pending legal-status Critical Current

Links

Images

Classifications

    • E—FIXED CONSTRUCTIONS
    • E02—HYDRAULIC ENGINEERING; FOUNDATIONS; SOIL SHIFTING
    • E02F—DREDGING; SOIL-SHIFTING
    • E02F9/00—Component parts of dredgers or soil-shifting machines, not restricted to one of the kinds covered by groups E02F3/00 - E02F7/00
    • E02F9/20—Drives; Control devices
    • E02F9/2025—Particular purposes of control systems not otherwise provided for
    • E02F9/205—Remotely operated machines, e.g. unmanned vehicles
    • E—FIXED CONSTRUCTIONS
    • E02—HYDRAULIC ENGINEERING; FOUNDATIONS; SOIL SHIFTING
    • E02F—DREDGING; SOIL-SHIFTING
    • E02F9/00—Component parts of dredgers or soil-shifting machines, not restricted to one of the kinds covered by groups E02F3/00 - E02F7/00
    • E02F9/26—Indicating devices
    • E02F9/261—Surveying the work-site to be treated
    • E—FIXED CONSTRUCTIONS
    • E02—HYDRAULIC ENGINEERING; FOUNDATIONS; SOIL SHIFTING
    • E02F—DREDGING; SOIL-SHIFTING
    • E02F3/00—Dredgers; Soil-shifting machines
    • E02F3/04—Dredgers; Soil-shifting machines mechanically-driven
    • E02F3/28—Dredgers; Soil-shifting machines mechanically-driven with digging tools mounted on a dipper- or bucket-arm, i.e. there is either one arm or a pair of arms, e.g. dippers, buckets
    • E02F3/36—Component parts
    • E02F3/42—Drives for dippers, buckets, dipper-arms or bucket-arms
    • E02F3/43—Control of dipper or bucket position; Control of sequence of drive operations
    • E02F3/435—Control of dipper or bucket position; Control of sequence of drive operations for dipper-arms, backhoes or the like

Definitions

  • the present invention relates to a technology of reporting a state of a work machine to a worker.
  • a remote operation assistance server of the present invention is
  • a server for assisting a remote operation of a work machine using a remote operation device comprises:
  • a first assistance processing element configured to acquire a picked-up image indicating an environment of the work machine through a real machine image pickup device loaded on the work machine, and make a work environment image according to the picked-up image be displayed at a remote image output device configuring the remote operation device;
  • a second assistance processing element configured to recognize a future change mode of a real space area projected in the work environment image displayed at the remote image output device by the first assistance processing element, and make a remote output interface configuring the remote operation device output a previous notice indicating the future change mode of the real space area.
  • the previous notice indicating the future change mode of the real space area projected in the work environment image displayed at the remote image output device is output by the remote output interface configuring the remote operation device.
  • a space occupancy mode of the real space area projected in the work environment image may be changed according to switching of one real machine image pickup device corresponding to the work environment image among the plurality of real machine image pickup devices and image processing by the first assistance processing element or the like in addition to an optical axis direction and/or a photographing magnification of the real machine image pickup device.
  • FIG. 1 is an explanatory drawing regarding a configuration of a remote operation assistance system.
  • FIG. 2 is an explanatory drawing regarding a configuration of a remote operation device.
  • FIG. 3 is an explanatory drawing regarding a configuration of a work machine.
  • FIG. 4 is an explanatory drawing regarding a function of the remote operation assistance system of a first embodiment.
  • FIG. 5 is an explanatory drawing regarding a work environment image.
  • FIG. 6 is an explanatory drawing regarding a first displacement mode of a virtual sound source.
  • FIG. 7 is an explanatory drawing regarding a second displacement mode of the virtual sound source.
  • FIG. 8 is an explanatory drawing regarding a function of a remote operation assistance system of a second embodiment.
  • a remote operation assistance system illustrated in FIG. 1 is configured by a remote operation assistance server 10 , a remote operation device 20 , and a work machine which can be remotely operated via the remote operation device 20 . Only one of the remote operation device 20 and the work machine 40 may be a component of the remote operation assistance system.
  • the remote operation assistance server 10 , the remote operation device 20 and the work machine 40 are configured to perform network communication with each other.
  • An intercommunication network of the remote operation assistance server 10 and the remote operation device 20 and an intercommunication network of the remote operation assistance server 10 and the work machine 40 may be the same or may be different.
  • the remote operation assistance server 10 comprises a database 102 , a first assistance processing element 121 , and a second assistance processing element 122 .
  • the database 102 stores and holds picked-up image data or the like.
  • the database 102 may be configured by a database server different from the remote operation assistance server 10 .
  • Each assistance processing element is configured by an arithmetic processing unit (a single-core processor, a multi-core processor, or a processor core configuring it), reads required data and software from a storage device such as a memory and executes arithmetic processing to be described later according to the software with the data as a target.
  • the remote operation device 20 comprises a remote controller 200 , a remote input interface 210 , and a remote output interface 220 .
  • the remote controller 200 is configured by an arithmetic processing unit (a single-core processor, a multi-core processor, or a processor core configuring it), reads required data and software from a storage device such as a memory and executes arithmetic processing according to the software with the data as a target.
  • arithmetic processing unit a single-core processor, a multi-core processor, or a processor core configuring it
  • the remote input interface 210 comprises a remote operation mechanism 211 .
  • the remote output interface 220 comprises a remote image output device 221 , a remote sound output device 222 , and a remote radio communication device 224 .
  • the remote operation mechanism 211 includes a traveling operation device, a turning operation device, a boom operation device, an arm operation device, and a bucket operation device.
  • Each operation device comprises an operation lever which receives a rotating operation.
  • the operation lever (traveling lever) of the traveling operation device is operated so as to move a lower traveling body 410 of the work machine 40 .
  • the traveling lever may also serve as a traveling pedal.
  • the traveling pedal fixed to a base part or a lower end part of the traveling lever may be provided.
  • the operation lever (turning lever) of the turning operation device is operated so as to move a hydraulic type turning motor configuring a turning mechanism 430 of the work machine 40 .
  • the operation lever (boom lever) of the boom operation device is operated so as to move a boom cylinder 442 of the work machine 40 .
  • the operation lever (arm lever) of the arm operation device is operated so as to move an arm cylinder 444 of the work machine 40 .
  • the operation lever (bucket lever) of the bucket operation device is operated so as to move a bucket cylinder 446 of the work machine 40 .
  • Each operation lever configuring the remote operation mechanism 211 is, for example, as illustrated in FIG. 2 , arranged around a seat St for an operator to sit.
  • the seat St is in a form like a high back chair with armrests, but may be a sitting part in an optional form that the operator can sit, such as a form like a low back chair without a headrest or a form like a chair without a backrest.
  • a pair of left and right traveling levers 2110 according to left and right crawlers are arranged laterally side by side at a front of the seat St.
  • One operation lever may serve as a plurality of operation levers.
  • a left side operation lever 2111 provided at the front of a left side frame of the seat St illustrated in FIG. 2 may function as the arm lever when operated in front and back directions and may function as the turning lever when operated in left and right directions.
  • a right side operation lever 2112 provided at the front of a right side frame of the seat St illustrated in FIG. 2 may function as the boom lever when operated in the front and back directions and may function as the bucket lever when operated in the left and right directions.
  • a lever pattern may be arbitrarily changed by an operation instruction of the operator.
  • the remote image output device 221 is configured by a center remote image output device 2210 , a left side remote image output device 2211 , and a right side remote image output device 2212 having a screen in a roughly rectangular shape arranged respectively at the front, diagonally left front, and diagonally right front of the seat St, as illustrated in FIG. 2 for example.
  • Shapes and sizes of the respective screens (image display areas) of the center remote image output device 2210 , the left side remote image output device 2211 , and the right side remote image output device 2212 may be the same or may be different.
  • a right edge of the left side remote image output device 2211 is adjacent to a left edge of the center remote image output device 2210 so that the screen of the center remote image output device 2210 and the screen of the left side remote image output device 2211 form an inclination angle ⁇ 1 (for example, 120° ⁇ 1 ⁇ 150°).
  • a left edge of the right side remote image output device 2212 is adjacent to a right edge of the center remote image output device 2210 so that the screen of the center remote image output device 2210 and the screen of the right side remote image output device 2212 form an inclination angle ⁇ 2 (for example, 120° ⁇ 2 ⁇ 150°).
  • the inclination angles ⁇ 1 and ⁇ 2 may be the same or may be different.
  • the respective screens of the center remote image output device 2210 , the left side remote image output device 2211 , and the right side remote image output device 2212 may be parallel to a vertical direction or may be inclined relative to the vertical direction.
  • At least one remote image output device of the center remote image output device 2210 , the left side remote image output device 2211 , and the right side remote image output device 2212 may be configured by multi-divisional remote image output devices.
  • the center remote image output device 2210 may be configured by a pair of vertically adjacent remote image output devices having the screen in the roughly rectangular shape.
  • the remote sound output device 222 is configured by a center sound output device 2220 , a left side sound output device 2221 , and a right side sound output device 2222 configured by one or more speakers and arranged respectively at the back of the seat St, a left armrest rear part, and a right armrest rear part, as illustrated in FIG. 2 for example. Respective specifications of the center sound output device 2220 , the left side sound output device 2221 , and the right side sound output device 2222 may be the same or may be different.
  • the remote sound output device 222 realizes a virtual sound source V ss which is sound capable of position displacement around the seat St
  • the work machine 40 comprises a real machine controller 400 , a real machine input interface 41 , a real machine output interface 42 , and a work mechanism 440 .
  • the real machine controller 400 comprises an image processor 30 .
  • the image processor 30 comprises a state detection element 31 , an image prediction element 32 and an image compression element 34 .
  • Each of the components of the real machine controller 400 and the image processor 30 is configured by an arithmetic processing unit (a single-core processor, a multi-core processor or a processor core configuring it), reads required data and software from a storage device such as a memory and executes arithmetic processing according to the software with the data as a target.
  • arithmetic processing unit a single-core processor, a multi-core processor or a processor core configuring it
  • the work machine 40 is a crawler shovel (construction machine) for example, and comprises, as illustrated in FIG. 3 , a crawler type lower traveling body 410 and an upper turning body 420 loaded on the lower traveling body 410 so as to be turnable through the turning mechanism 430 .
  • a cab 424 (driving room) is provided on a front left side part of the upper turning body 420 .
  • the work mechanism 440 is provided on a front center part of the upper turning body 420 .
  • the real machine input interface 41 comprises a real machine operation mechanism 411 , a real machine image pickup device 412 , and a real machine periphery monitoring device 414 .
  • the real machine operation mechanism 411 comprises a plurality of operation levers arranged similarly to the remote operation mechanism 211 around the seat arranged inside the cab 424 .
  • a driving mechanism or a robot which receives a signal according to an operation mode of a remote operation lever and moves a real machine operation lever based on the received signal is provided in the cab 424 .
  • the real machine image pickup device 412 is installed inside the cab 422 for example, and picks up an image of an environment including at least a part of the work mechanism 440 over a front window and a pair of left and right side windows.
  • the real machine periphery monitoring device 414 is configured by a ranging sensor or the like for detecting presence of a target object (for example, a worker, another work machine, and/or a vehicle) in a real space area (for example, a right side area, a left side area and/or a rear area and/or a dead angle area of the cab 424 ) where the image cannot be picked up by the real machine image pickup device 412 in a basic posture (for example, the posture in which an optical axis is turned to the front).
  • a target object for example, a worker, another work machine, and/or a vehicle
  • a real space area for example, a right side area, a left side area and/or a rear area and/or a dead angle area of the cab 424
  • a basic posture for example, the posture in which an optical axis is turned to the front.
  • the real machine output interface 42 comprises a real machine radio communication device 422 .
  • the work mechanism 440 as a work mechanism comprises a boom 441 mounted on the upper turning body 420 so as to be hoistable, an arm 443 rotatably connected to a distal end of the boom 441 , and a bucket 445 rotatably connected to a distal end of the arm 443 .
  • the boom cylinder 442 , the arm cylinder 444 , and the bucket cylinder 446 configured by extendable hydraulic cylinders are mounted.
  • the boom cylinder 442 is interposed between the boom 441 and the upper turning body 420 so as to be extended and contracted by receiving supply of hydraulic oil and rotate the boom 441 in a hoisting direction.
  • the arm cylinder 444 is interposed between the arm 443 and the boom 441 so as to be extended and contracted by receiving the supply of the hydraulic oil and rotate the arm 443 around a horizontal axis to the boom 441 .
  • the bucket cylinder 446 is interposed between the bucket 445 and the arm 443 so as to be extended and contracted by receiving the supply of the hydraulic oil and rotate the bucket 445 around the horizontal axis to the arm 443 .
  • FIG. 4 is a flowchart explaining the function of the remote operation assistance system of the configuration described above.
  • a block “C” is used to simplify description, means transmission and/or reception of data and means a conditional branch of executing processing in a branch direction on condition that the data is transmitted and/or received.
  • an environment checking request is transmitted to the remote operation assistance server 10 via the remote radio communication device 224 ( FIG. 4 /STEP 210 ).
  • a specifying operation performed via the remote input interface 210 by an operator may be determined as a transmission start requirement for the environment checking request.
  • the “specifying operation” is an operation of tapping or the like in the remote input interface 210 for specifying the work machine 40 that the operator intends to remotely control, for example.
  • the environment checking request is transmitted to the pertinent work machine 40 by the first assistance processing element 121 ( FIG. 4 /C 10 ).
  • a picked-up image is acquired via the real machine image pickup device 412 by the real machine controller 400 , and picked-up image data indicating the picked-up image is transmitted to the remote operation assistance server 10 via the real machine radio communication device 422 ( FIG. 4 /STEP 410 ).
  • the remote operation assistance server 10 when the picked-up image data is received by the first assistance processing element 121 ( FIG. 4 /C 11 ), work environment image data according to the picked-up image is transmitted to the remote operation device by the second assistance processing element 122 ( FIG. 4 /STEP 110 ).
  • the work environment image data is, in addition to the picked-up image data itself, image data indicating a simulated work environment image generated based on the picked-up image.
  • the remote operation device 20 when the work environment image data is received via the remote radio communication device 224 ( FIG. 4 /C 21 ), a work environment image according to the work environment image data is output to the remote image output device 221 by the remote controller 200 ( FIG. 4 /STEP 212 ).
  • the work environment image projecting the boom 441 and the arm 443 as a part of the work mechanism 440 and a heap of rubble or soil (which is a work target by the bucket 445 ) in front of the cab 424 through a window frame demarcating the cab 424 is output to the remote image output device 221 (in particular, the center remote image output device 2210 ).
  • the real machine controller 400 determines whether or not the work machine 40 is in a specified state ( FIG. 4 /STEP 411 ). At the time, which specified state of the plurality of predetermined different specified states the specified state is pertinent to may be determined.
  • the “specified state” is a state where the real machine controller 400 is set or programmed beforehand so as to make the real machine image pickup device 412 demonstrate the pan function and/or tilt function and the zoom function as the work machine 40 is turned to the specified state.
  • a state where the presence of the target object is detected in the dead angle area of the real machine image pickup device 412 in the normal posture by the real machine periphery monitoring device 414 may be defined as the specified state.
  • the left side area of the cab 424 is the dead angle area of the real machine image pickup device 412 in the normal posture, and when the presence of the target object is detected in the left side area, the optical axis is turned to a left direction by the pan function of the real machine image pickup device 412 and the optical axis is turned downward (or upward) by the tilt function as needed.
  • the right side area of the cab 424 is the dead angle area of the real machine image pickup device 412 in the normal posture, and when the presence of the target object is detected in the right side area, the optical axis is turned to a right direction by the pan function of the real machine image pickup device 412 and the optical axis is turned downward (or upward) by the tilt function as needed.
  • a state where the presence of the target object of a specified kind is detected in the real space area to be an image pickup target of the real machine image pickup device 412 may be defined as the specified state.
  • the target object for example, a human such as a worker
  • a zoom-in function of the real machine image pickup device 412 a display magnification of an image area including the target object is increased.
  • the presence of the target object for example, another work machine
  • a zoom-out function of the real machine image pickup device 412 the display magnification of the image area including the target object is reduced.
  • the pan function and/or the tilt function may be controlled in order to adjust a position of the target object in the picked-up image.
  • a state where it is detected that a displacement amount and/or a displacement speed of the target object is a predetermined value or greater in the real space area to be the image pickup target of the real machine image pickup device 412 may be defined as the specified state.
  • the displacement amount of the target object for example, the bucket 445
  • the tilt function of the real machine image pickup device 412 exceeds the predetermined value, by the pan function and/or the tilt function of the real machine image pickup device 412 , the direction of the optical axis of the real machine image pickup device 412 is changed so as to follow the target object.
  • a control mode of a posture change function (the pan function and/or the tilt function) and/or the zoom function of the real machine image pickup device 412 according to the specified state is recognized, and the recognition result is transmitted to the remote operation assistance server 10 via the real machine radio communication device 422 ( FIG. 4 /STEP 412 ).
  • a correspondence relation between (the kind of) the specified state and the control mode of the posture change function and/or the zoom function of the real machine image pickup device 412 may be stored in a storage device configuring the real machine controller 400 or the database 102 .
  • the second assistance processing element 122 when the control mode of the posture change function and/or the zoom function of the real machine image pickup device 412 is received ( FIG. 4 /C 12 ), by the second assistance processing element 122 , a change mode of the real space area projected in the picked-up image (consequently the work environment image) is recognized and then transmitted to the work machine 40 ( FIG. 4 /STEP 112 ). In the case where the direction of the optical axis is changed to the left and right by the pan function of the real machine image pickup device 412 , by the second assistance processing element 122 , it is recognized that the real space area projected in the picked-up image is similarly displaced to the left and right.
  • the remote controller 200 when the change mode of the real space area is received via the remote radio communication device 224 ( FIG. 4 /C 22 ), by the remote controller 200 , a previous notice indicating the change mode is output via the remote output interface 220 ( FIG. 4 /STEP 214 ).
  • the virtual sound source V ss realized by the remote sound output device 222 is controlled so as to be displaced in the left direction (the same direction as the displacement direction of the real space area).
  • a graphic or icon A 1 in an arrow shape in the left direction may be output to the remote image output device 221 .
  • a specified icon may be output so as to be moved in the same direction as the virtual sound source V ss in the remote image output device 221 .
  • control is performed so that the sound gradually becomes strong and/or a frequency of the sound gradually becomes high in a state where the virtual sound source V ss realized by the remote sound output device 222 is localized.
  • a graphic or icon A 2 in the arrow shape toward an image area S enlarged by zoom-in and/or an outer side of the image area S may be output to the remote image output device 221 .
  • a specified icon may be output so as to be scaled according to scaling of the real space area or the image area in the remote image output device 221 .
  • the control is performed so that the sound gradually becomes weak and/or the frequency of the sound gradually becomes low in the state where the virtual sound source V ss realized by the remote sound output device 222 is localized.
  • the graphic or icon A 2 in the arrow shape toward the image area reduced by zoom-out and/or an inner side of the image area may be output to the remote image output device 221 .
  • a previous notice output completion report indicating that the previous notice has been output via the remote output interface 220 is transmitted to the remote operation assistance server 10 via the remote radio communication device 224 ( FIG. 4 /STEP 216 ).
  • the remote operation assistance server 10 when the previous notice output completion report is received, the previous notice output completion report is transmitted to the work machine 40 by the second assistance processing element 122 ( FIG. 4 /C 14 ).
  • the posture change function and/or the zoom function of the real machine image pickup device 412 is controlled in a form according to the specified state by the real machine controller 400 ( FIG. 4 /STEP 414 ).
  • the real space area projected in the work environment image is changed from a front area of the work machine 40 to a diagonally left front area.
  • the optical axis direction is changed from a front horizontal direction to a diagonally lower front by the control of the tilt function of the real machine image pickup device 412 , the real space area projected in the work environment image is changed to an area lower than before.
  • the real space area projected in the work environment image is reduced (or the image area is enlarged).
  • the zoom-out function of the real machine image pickup device 412 and increasing the zoom magnification the real space area projected in the work environment image is enlarged (or the image area is reduced).
  • an operation mode of the remote operation mechanism 211 is recognized by the remote controller 200 , and a remote operation command according to the operation mode is transmitted to the remote operation assistance server 10 via the remote radio communication device 224 ( FIG. 4 /STEP 220 ).
  • the remote operation assistance server 10 when the remote operation command is received by the second assistance processing element 122 , the remote operation command is transmitted to the work machine 40 by the first assistance processing element 121 ( FIG. 4 /C 16 ).
  • the previous notice indicating a future change mode of the rear space area projected in the work environment image displayed at the remote image output device 221 is output to the remote output interface 220 .
  • the operator who encounters the virtual sound source V ss which emits the sound for which the strength and/or the height of the frequency is changed according to a scaling mode of the real space area and/or the icon A 2 in the arrow shape indicating the scaling mode of the real space area recognize beforehand that the real space area projected in the work environment image is to be scaled according to the scaling mode in the future (see FIG. 7 ).
  • the space occupancy mode of the real space area projected in the work environment image may be enlarged or reduced according to the change of a photographing magnification of the real machine image pickup device 412 . Therefore, the mental burdens on the operator when the real space area projected in the work environment image displayed at the remote image output device 221 is enlarged or reduced as the previous notice are reduced.
  • the first assistance processing element 121 and the second assistance processing element 122 are configured by the remote operation assistance server 10 , however, as another embodiment, the first assistance processing element 121 and/or the second assistance processing element 122 may be configured by the work machine 40 and/or the remote operation device 20 .
  • control mode of the posture change function and/or the zoom function of the real machine image pickup device 412 according to the specified state is recognized by the real machine controller 400 as it is recognized that the work machine 40 is in the specified state (see FIG. 4 /STEP 411 ⁇ YES STEP 412 ), however, as another embodiment, the control mode of the posture change function and/or the zoom function of the real machine image pickup device 412 according to the specified state may be recognized by the remote operation assistance server 10 or the second assistance processing element 122 .
  • FIG. 8 a series of processing may be executed according to a flowchart illustrated in FIG. 8 instead of the flowchart illustrated in FIG. 4 .
  • the block “C” is used to simplify the description, means the transmission and/or the reception of the data and means the conditional branch of executing the processing in the branch direction on condition that the data is transmitted and/or received.
  • FIG. 8 for the processing in common with FIG. 4 , a same sign is used and the explanation is omitted.
  • the specified state is transmitted to the remote operation assistance server 10 via the real machine radio communication device 422 ( FIG. 8 /STEP 413 ).
  • the second assistance processing element 122 when the specified state is received ( FIG. 8 /C 13 ), by the second assistance processing element 122 , the control mode of the posture change function (the pan function and/or the tilt function) and/or the zoom function of the real machine image pickup device 412 according to the specified state is recognized ( FIG. 8 /STEP 111 ). Further, by the second assistance processing element 122 , the change mode of the real space area projected in the picked-up image (consequently the work environment image) according to the control mode of the posture change function and/or the zoom function of the real machine image pickup device 412 is recognized and then transmitted to the work machine 40 ( FIG. 8 /STEP 112 ).
  • a control command for the posture change function and/or the zoom function of the real machine image pickup device 412 is generated in the form according to the specified state, and is transmitted to the work machine 40 ( FIG. 8 /STEP 114 ).
  • the posture change function and/or the zoom function of the real machine image pickup device 412 is controlled according to the control command ( FIG. 8 /STEP 414 ).
  • the control mode of the posture change function and/or the zoom function of the real machine image pickup device 412 according to the specified state is recognized by the real machine controller 400 as it is recognized that the work machine 40 is in the specified state (see FIG. 4 /STEP 411 . . . YES SIFT 412 ), however, as another embodiment, the control mode of the posture change function and/or the zoom function of the real machine image pickup device 412 according to the specified state may be recognized by the remote operation device 20 or the remote controller 200 .
  • the real space area projected in the picked-up image and the corresponding work environment image is continuously displaced by the pan function and/or the tilt function of the real machine image pickup device 412
  • the plurality of real machine image pickup devices having different image pickup areas may be loaded on the work machine 40 and the real space area projected in the work environment image may be discontinuously displaced by switching the real machine image pickup device which acquires the picked-up image to be a base of the work environment image from one real machine image pickup device to another real machine image pickup device.
  • one real machine image pickup device may be switched to another real machine image pickup device the image pickup area of which is the real space area where the target object is detected by the real machine periphery monitoring device 414 .
  • the remote output interface 220 the mental burdens put on the operator are reduced even when the work environment image output at the remote image output device 221 is discontinuously displaced or changed.
  • the real space area projected in the picked-up image and the corresponding work environment image is continuously displaced by the pan function and/or the tilt function of the real machine image pickup device 412 , however, as another embodiment, the real space area projected in the work environment image may be displaced and/or scaled by image processing when the work environment image is generated by the first assistance processing element 121 . Even in this case, by the previous notice output by the remote output interface 220 , the mental burdens put on the operator are reduced even when the work environment image output at the remote image output device 221 is displaced and/or scaled.
  • the zoom-in and the zoom-out of the real machine image pickup device 412 are previously noticed by the strength (the height of a sound pressure) of the sound emitted from the virtual sound source V ss and/or the height of the frequency (see FIG. 7 ).
  • the real machine image pickup device 412 may have only the pan function, and the zoom-in and the zoom-out of the real machine image pickup device 412 may be previously noticed by the up and down displacement of the virtual sound source V ss before and after the change of a pan angle is previously noticed by the left and right displacement of the virtual sound source V ss (see FIG. 6 ).
  • the real machine image pickup device 412 may have only the tilt function, and the zoom-in and the zoom-out of the real machine image pickup device 412 may be previously noticed by the left and right displacement of the virtual sound source V ss before and after the change of a tilt angle is previously noticed by the up and down displacement of the virtual sound source V ss .
  • the second assistance processing element recognizes the future change mode of the real space area projected in the work environment image, according to the specified state detected by the real machine periphery monitoring device loaded on the work machine.
  • the remote operation assistance server of the configuration it is possible to make the operator who receives the previous notice of the form of the displacement of an output part recognize beforehand that the real space area projected in the work environment image is to be displaced according to the displacement mode in the future and also the work machine is in the specified state. Therefore, the mental burdens on the operator when the real space area projected in the work environment image displayed at the remote image output device is displaced as the previous notice are reduced.
  • the second assistance processing element recognizes the future displacement mode of the real space area as the future change mode of the real space area, and makes the output part of the previous notice configured by the remote output interface be displaced according to the future displacement mode of the real space area.
  • the remote operation assistance server of the configuration it is possible to make the operator who receives the previous notice of the form of the displacement of the output part recognize beforehand that the real space area projected in the work environment image is to be displaced according to the displacement mode in the future.
  • the space occupancy mode of the real space area projected in the work environment image may be displaced according to the change of the optical axial direction of the real machine image pickup device, switching of one real machine image pickup device corresponding to the work environment image among the plurality of real machine image pickup devices and the image processing by the first assistance processing element or the like. Therefore, the mental burdens on the operator when the real space area projected in the work environment image displayed at the remote image output device is displaced as the previous notice are reduced.
  • the second assistance processing element further recognizes a future scaling mode of the real space area as the future change mode of the real space area, and makes the remote output interface displace the output part of the previous notice according to the future scaling mode of the real space area before or after making the output part of the previous notice configured by the remote output interface be displaced according to the future displacement mode of the real space area.
  • the remote operation assistance server of the configuration it is possible to make the operator who receives the previous notice of the form of the displacement of the output part over two stages recognize beforehand that the real space area projected in the work environment image is to be displaced according to a first displacement mode and is to be scaled according to a second displacement mode in the future. Therefore, the mental burdens on the operator when the real space area projected in the work environment image displayed at the remote image output device is displaced and scaled as the previous notice are reduced.
  • the second assistance processing element makes the remote sound output device configuring the remote output interface displace the virtual sound source that emits the sound as the previous notice.
  • the remote operation assistance server of the configuration it is possible to make the operator who receives the previous notice of the form of the displacement of the virtual sound source recognize beforehand that the real space area projected in the work environment image is to be displaced (and is to be scaled further) according to the displacement mode in the future. For example, when the virtual sound source is displaced to right (left, up and down), it is possible to make the operator recognize that the real space area projected in the work environment image is to be displaced to right (left, up and down) immediately after. At the time, since the previous notice is reported to the operator not through visual sensation but through auditory sensation, a situation of distracting a visual attention of the operator to the work environment image is avoided. Therefore, the mental burdens on the operator when the real space area projected in the work environment image displayed at the remote image output device is displaced as the previous notice are reduced.
  • the second assistance processing element recognizes the future displacement mode of the real space area as the future change mode of the real space area according to a switching mode from one real machine image pickup device of the plurality of real machine image pickup devices loaded on the work machine to another real machine image pickup device.
  • the remote operation assistance server of the configuration it is possible to make the operator who receives the previous notice of the change of the output mode recognize beforehand that the future scaling mode of the real space area projected in the work environment image is to be changed according to the change of the output mode resulting from switching of the real machine image pickup device. Therefore, the mental burdens on the operator when the real space area projected in the work environment image displayed at the remote image output device is enlarged or reduced as the previous notice are reduced.
  • the second assistance processing element recognizes the future scaling mode of the real space area as the future change mode of the real space area, and makes the remote output interface change the output mode of the previous notice according to the future scaling mode of the real space area.
  • the remote operation assistance server of the configuration it is possible to make the operator who receives the previous notice of the change of the output mode recognize beforehand that the future scaling mode of the real space area projected in the work environment image is to be changed according to the change of the output mode.
  • the space occupancy mode of the real space area projected in the work environment image may be enlarged or reduced according to the photographing magnification of the real machine image pickup device, the switching of one real machine image pickup device corresponding to the work environment image among the plurality of real machine image pickup devices and the image processing by the first assistance processing element or the like. Therefore, the mental burdens on the operator when the real space area projected in the work environment image displayed at the remote image output device is enlarged or reduced as the previous notice are reduced.
  • the second assistance processing element makes the remote sound output device configuring the remote output interface change at least one of the height of a sound pressure and the height of the frequency of the sound as the previous notice.
  • the remote operation assistance server of the configuration it is possible to make the operator who receives the previous notice as the sound for which the height of the sound pressure and/or the height of the frequency is changed recognize beforehand that the real space area projected in the work environment image is to be scaled according to the change mode in the future. For example, when the sound pressure gradually becomes high, it is possible to make the operator recognize beforehand that the real space area is to be reduced. On the other hand, when the sound pressure gradually becomes low, it is possible to make the operator recognize beforehand that the real space area is to be enlarged. At the time, since the previous notice is reported to the operator not through the visual sensation but through the auditory sensation, the situation of distracting the visual attention of the operator to the work environment image is avoided. Therefore, the mental burdens on the operator when the real space area projected in the work environment image displayed at the remote image output device is enlarged or reduced as the previous notice are reduced.

Landscapes

  • Engineering & Computer Science (AREA)
  • Mining & Mineral Resources (AREA)
  • Civil Engineering (AREA)
  • General Engineering & Computer Science (AREA)
  • Structural Engineering (AREA)
  • Component Parts Of Construction Machinery (AREA)
  • Manipulator (AREA)
  • User Interface Of Digital Computer (AREA)
US18/278,209 2021-03-19 2021-12-13 Remote operation assistance server and remote operation asistance system Pending US20240141616A1 (en)

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
JP2021045675A JP2022144601A (ja) 2021-03-19 2021-03-19 遠隔操作支援サーバおよび遠隔操作支援システム
JP2021-045675 2021-03-19
PCT/JP2021/045780 WO2022195988A1 (fr) 2021-03-19 2021-12-13 Serveur d'assistance à la téléopération et système d'assistance à la téléopération

Publications (1)

Publication Number Publication Date
US20240141616A1 true US20240141616A1 (en) 2024-05-02

Family

ID=83320041

Family Applications (1)

Application Number Title Priority Date Filing Date
US18/278,209 Pending US20240141616A1 (en) 2021-03-19 2021-12-13 Remote operation assistance server and remote operation asistance system

Country Status (5)

Country Link
US (1) US20240141616A1 (fr)
EP (1) EP4279667A1 (fr)
JP (1) JP2022144601A (fr)
CN (1) CN117043732A (fr)
WO (1) WO2022195988A1 (fr)

Family Cites Families (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2005286590A (ja) * 2004-03-29 2005-10-13 Fuji Photo Film Co Ltd 画像表示方法および画像表示装置
JP2008294724A (ja) * 2007-05-24 2008-12-04 Panasonic Corp 画像受信装置
JP5902444B2 (ja) * 2011-11-24 2016-04-13 京セラ株式会社 携帯端末装置、プログラムおよび表示制御方法
WO2016174754A1 (fr) * 2015-04-28 2016-11-03 株式会社小松製作所 Dispositif de surveillance de la périphérie pour engin de chantier et procédé de surveillance de la périphérie pour engin de chantier
JP6581139B2 (ja) 2017-03-31 2019-09-25 日立建機株式会社 作業機械の周囲監視装置
JP2019156533A (ja) 2018-03-09 2019-09-19 株式会社タダノ クレーン
JP7151392B2 (ja) * 2018-11-07 2022-10-12 コベルコ建機株式会社 建設機械の遠隔操作装置
JP7310338B2 (ja) * 2019-06-10 2023-07-19 コベルコ建機株式会社 遠隔操作システムおよび遠隔操作サーバ

Also Published As

Publication number Publication date
CN117043732A (zh) 2023-11-10
WO2022195988A1 (fr) 2022-09-22
EP4279667A1 (fr) 2023-11-22
JP2022144601A (ja) 2022-10-03

Similar Documents

Publication Publication Date Title
JP7108750B2 (ja) システムおよび方法
US11447928B2 (en) Display system, display method, and remote control system
US20140111648A1 (en) Device For Monitoring Area Around Working Machine
JP6917167B2 (ja) 建設機械用俯瞰画像表示装置
US11993922B2 (en) Remote operation system
CN108432241B (zh) 挖土机
US20230073325A1 (en) Remote operation assistance server, remote operation assistance system, and remote operation assistance method
US20240141616A1 (en) Remote operation assistance server and remote operation asistance system
EP4130403A1 (fr) Dispositif et procédé de traitement d'image
JP6901710B1 (ja) 安全装置および作業機械
CN114868382A (zh) 作业辅助服务器、拍摄装置的选择方法
US20230056724A1 (en) Remote operation device, remote operation assistance server, remote operation assistance system, and remote operation assistance method
US11939744B2 (en) Display system, remote operation system, and display method
US20220303472A1 (en) Image capturing function control system and image capturing function control method
CN115362295B (zh) 远程操作辅助服务器、远程操作辅助系统以及远程操作辅助方法
EP4060129A1 (fr) Système de prise en charge d'opération à distance et procédé de prise en charge d'opération à distance
CN113615164B (zh) 远程操作系统以及远程操作服务器
US20230003002A1 (en) Remote operation device and remote operation system
WO2023091383A1 (fr) Système d'évitement de collision et procédé pour éviter une collision d'engin de chantier avec des obstacles
CN115362297A (zh) 工程机械的周边检测装置

Legal Events

Date Code Title Description
AS Assignment

Owner name: KOBELCO CONSTRUCTION MACHINERY CO., LTD., JAPAN

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:MORITA, YUKI;OTANI, MASAKI;SAWADA, YUSUKE;AND OTHERS;REEL/FRAME:064661/0637

Effective date: 20230615

STPP Information on status: patent application and granting procedure in general

Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION