EP4079977A1 - Système de contrôle à distance - Google Patents

Système de contrôle à distance Download PDF

Info

Publication number
EP4079977A1
EP4079977A1 EP22160221.2A EP22160221A EP4079977A1 EP 4079977 A1 EP4079977 A1 EP 4079977A1 EP 22160221 A EP22160221 A EP 22160221A EP 4079977 A1 EP4079977 A1 EP 4079977A1
Authority
EP
European Patent Office
Prior art keywords
work machine
remote operation
work
remote
machine
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
EP22160221.2A
Other languages
German (de)
English (en)
Other versions
EP4079977B1 (fr
Inventor
Ryuichi Hirose
Yuki Morita
Yuya Matsuda
Masaki Otani
Seiji Saiki
Yoichiro Yamazaki
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Kobelco Construction Machinery Co Ltd
Original Assignee
Kobelco Construction Machinery Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Kobelco Construction Machinery Co Ltd filed Critical Kobelco Construction Machinery Co Ltd
Publication of EP4079977A1 publication Critical patent/EP4079977A1/fr
Application granted granted Critical
Publication of EP4079977B1 publication Critical patent/EP4079977B1/fr
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/12—Protocols specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
    • H04L67/125—Protocols specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks involving control of end-device applications over a network
    • E—FIXED CONSTRUCTIONS
    • E02—HYDRAULIC ENGINEERING; FOUNDATIONS; SOIL SHIFTING
    • E02F—DREDGING; SOIL-SHIFTING
    • E02F9/00—Component parts of dredgers or soil-shifting machines, not restricted to one of the kinds covered by groups E02F3/00 - E02F7/00
    • E02F9/20—Drives; Control devices
    • E—FIXED CONSTRUCTIONS
    • E02—HYDRAULIC ENGINEERING; FOUNDATIONS; SOIL SHIFTING
    • E02F—DREDGING; SOIL-SHIFTING
    • E02F9/00—Component parts of dredgers or soil-shifting machines, not restricted to one of the kinds covered by groups E02F3/00 - E02F7/00
    • E02F9/20—Drives; Control devices
    • E02F9/2025—Particular purposes of control systems not otherwise provided for
    • E02F9/205—Remotely operated machines, e.g. unmanned vehicles
    • E—FIXED CONSTRUCTIONS
    • E02—HYDRAULIC ENGINEERING; FOUNDATIONS; SOIL SHIFTING
    • E02F—DREDGING; SOIL-SHIFTING
    • E02F9/00—Component parts of dredgers or soil-shifting machines, not restricted to one of the kinds covered by groups E02F3/00 - E02F7/00
    • E02F9/20—Drives; Control devices
    • E02F9/2025—Particular purposes of control systems not otherwise provided for
    • E02F9/2054—Fleet management
    • E—FIXED CONSTRUCTIONS
    • E02—HYDRAULIC ENGINEERING; FOUNDATIONS; SOIL SHIFTING
    • E02F—DREDGING; SOIL-SHIFTING
    • E02F9/00—Component parts of dredgers or soil-shifting machines, not restricted to one of the kinds covered by groups E02F3/00 - E02F7/00
    • E02F9/26—Indicating devices
    • E—FIXED CONSTRUCTIONS
    • E02—HYDRAULIC ENGINEERING; FOUNDATIONS; SOIL SHIFTING
    • E02F—DREDGING; SOIL-SHIFTING
    • E02F9/00—Component parts of dredgers or soil-shifting machines, not restricted to one of the kinds covered by groups E02F3/00 - E02F7/00
    • E02F9/26—Indicating devices
    • E02F9/264—Sensors and their calibration for indicating the position of the work tool
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
    • G05D1/0011—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots associated with a remote control arrangement
    • G05D1/0027—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots associated with a remote control arrangement involving a plurality of vehicles, e.g. fleet or convoy travelling
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
    • G05D1/0011—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots associated with a remote control arrangement
    • G05D1/0038—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots associated with a remote control arrangement by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06K—GRAPHICAL DATA READING; PRESENTATION OF DATA; RECORD CARRIERS; HANDLING RECORD CARRIERS
    • G06K17/00—Methods or arrangements for effecting co-operative working between equipments covered by two or more of main groups G06K1/00 - G06K15/00, e.g. automatic card files incorporating conveying and reading operations
    • G06K17/0022—Methods or arrangements for effecting co-operative working between equipments covered by two or more of main groups G06K1/00 - G06K15/00, e.g. automatic card files incorporating conveying and reading operations arrangements or provisions for transferring data to distant stations, e.g. from a sensing device
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
    • H04N7/181—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast for receiving images from a plurality of remote sources
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Definitions

  • the present invention relates to a remote operation system having a mutual communication function of communicating with each of plural work machines and with a remote operation device for remotely operating one work machine as a remote operation target among the plural work machines.
  • Japanese Patent Laid-Open No. 2019-176401 discloses a remote operation system in which a master and a slave are provided to be capable of communicating with each other, the operator operates an operation lever provided to the master, and the slave arranged in an operator seat of a work machine thereby operates an operation lever of the work machine.
  • the master and respective slaves provided to work machines have to be connected together to be capable of communicating with each other.
  • an operator performs a remote operation by operating an operation lever or the like while viewing an image around the work machine as an operation target by a monitor of the master.
  • an image displayed on the monitor of the master is an image around a work machine which is different from the work machine as the operation target.
  • the operator can check an external appearance of the work machine and circumstances of a site where the work machine is positioned, but in a case where the work machine is remotely operated, such a check cannot be performed.
  • a remote operation device is connected with an unintended work machine (different slave).
  • the present invention has been made in consideration of such problems, and an object thereof is to provide a remote operation system that is capable of making a check about whether a work machine which acts in accordance with an operation by an operator agrees with a work machine selected by the operator.
  • a remote operation system of the present invention is a remote operation system having a mutual communication function of communicating with each of plural work machines and with a remote operation device which remotely operates a cooperating work machine as one work machine as a remote operation target among the plural work machines, the remote operation system including: a first support process element that acquires a determination result about whether or not a designation operation of selection information specifying the cooperating work machine as the remote operation target among the plural work machines in the remote operation device is executed and acquires a captured image which is captured by an actual machine image-capturing device mounted on a first work machine establishing communication with the remote operation device by a first communication path associated with the selection information in a case where the selection information is selected in the remote operation device and that causes an output interface of the remote operation device to output the captured image; and a second support process element that recognizes that one designation operation is performed, which is a portion of plural kinds of operations for causing a second work machine among the plural work machines to act, the second work machine establishing communication with the remote operation device by a
  • a work support system as one embodiment of the present invention is configured with a work support server 10 for supporting remote operations of a first selected work machine 40 and a second selected work machine 50 by a remote operation device 20.
  • the work support server 10, the remote operation device 20, the first selected work machine 40, and the second selected work machine 50 are configured to be capable of communicating with each other via a shared or separate network.
  • the work support server 10 is capable of being connected with the first selected work machine 40 or the second selected work machine 50, but it is sufficient that the work support server 10 is capable of being connected with two or more work machines, and the number of connected work machines is not limited.
  • either one of the first selected work machine 40 or the second selected work machine 50 which are operated by the remote operation device 20 will also be referred to as a cooperating work machine.
  • the work support server 10 will be described as a remote operation system which has a mutual communication function of communicating with each of plural selected work machines and with a remote operation device for remotely operating a cooperating work machine as a remote operation target among the plural selected work machines. Further, in the present embodiment, a description will be made about an example of a work support system which configures a composite system with the work support server 10, the remote operation device 20, the first selected work machine 40, and the second selected work machine 50.
  • “Acquiring" of various kinds of information by configuration elements (hardware) of the present invention represents a concept encompassing all computation processes for preparing various kinds of information in usable forms in the following computation processes, such as receiving the information, reading or searching for the designation information from an internal storage device (for example, a memory) and/or an external storage device (for example, an external database server), and calculating, estimating, predicting, identifying, and so forth designation information by executing a computation process for information, which is received, read, searched for, and so forth, as a target.
  • an internal storage device for example, a memory
  • an external storage device for example, an external database server
  • the work support server 10 comprises a database 102, a first support process element 121, and a second support process element 122.
  • the database 102 stores and retains captured image data and so forth.
  • the database 102 may be configured with a database server separate from the work support server 10.
  • the database 102 retains first identifiers (an identifier A-1, an identifier B-1, and an identifier C-1) which are assigned to communication paths (first communication path) of an actual machine image-capturing device 412 mounted on the first selected work machine 40 and of an actual machine image-capturing device 512 mounted on the second selected work machine 50 and which are for identifying work machines on a network
  • first identifier is a first IP address assigned to each apparatus of a work machine, for example
  • second identifiers an identifier A-2, an identifier B-2, and an identifier C-2 which are assigned to communication paths (second communication path) of an operation system of the first selected work machine 40 and of an operation system of the second selected work machine 50 and which are for identifying work machines on a network
  • the second identifier is a second IP address assigned to each apparatus of a work machine, for example
  • actual machine identification information which includes information of the respective work machines corresponding to the identifiers assigned to the first selected work machine 40
  • the database 102 stores map data (not illustrated) of a region including each piece of position information.
  • map data plural sets of map data stored in the database 102 are desirably stored while being sorted by work sites.
  • the plural sets of map data are desirably stored such that the map data corresponding to a work site selected by the operator are capable of being referred to.
  • Each of the first support process element 121 and the second support process element 122 is configured with an arithmetic processing unit (a single-core processor or a multi-core processor or a processor core configuring that), reads necessary data and software from a storage device such as a memory, and executes a computation process, described later, following the software for the data as a target.
  • an arithmetic processing unit a single-core processor or a multi-core processor or a processor core configuring that
  • the remote operation device 20 comprises a remote control device 200, a remote input interface 210, and a remote output interface 220.
  • the remote control device 200 is configured with an arithmetic processing unit (a single-core processor or a multi-core processor or a processor core configuring that), reads necessary data and software from a storage device such as a memory, and executes a computation process following the software for the data as a target.
  • arithmetic processing unit a single-core processor or a multi-core processor or a processor core configuring that
  • the remote input interface 210 comprises a remote operation mechanism 211 and a hand terminal 2240.
  • the remote output interface 220 comprises a remote image output device 221, a remote sound output device 222, a remote wireless communication apparatus 224, and the hand terminal 2240.
  • the hand terminal 2240 is used for both of the remote input interface 210 and the remote output interface 220.
  • the remote operation mechanism 211 includes operation devices for one designation operation, each of which is for performing one designation operation, such as a wiper operation device for a wiper which wipes water drops such as rain water adhered on an outer surface of a windshield of the first selected work machine 40 or the second selected work machine 50 and a headlight operation device for a headlight which is capable of projecting light to a front area of the first selected work machine 40 or the second selected work machine 50.
  • operation devices for one designation operation each of which is for performing one designation operation, such as a wiper operation device for a wiper which wipes water drops such as rain water adhered on an outer surface of a windshield of the first selected work machine 40 or the second selected work machine 50 and a headlight operation device for a headlight which is capable of projecting light to a front area of the first selected work machine 40 or the second selected work machine 50.
  • one designation operation preferably assists work of a work machine.
  • One designation operation is preferably about an operation such as an operation of a wiper device or the like and an operation for a lighting device such as a headlight, a stop lamp, or a room light, for example.
  • a lighting device such as a headlight, a stop lamp, or a room light, for example.
  • one designation operation may be about an operation for a sound output device which outputs a buzzer or an activation sound.
  • the operation device for one designation operation has a push button which accepts an input by depression, for example.
  • the wiper operation device (push button) is depressed and thereby starts or stops an action of the wiper of the first selected work machine 40 or the second selected work machine 50.
  • the headlight operation device (push button) is depressed and thereby starts or stops lighting by the headlight of the first selected work machine 40 or the second selected work machine 50.
  • the remote operation mechanism 211 includes a traveling operation device which accepts an operation about traveling of the first selected work machine 40 or the second selected work machine 50, a revolution operation device which accepts an operation about revolution of the first selected work machine 40 or the second selected work machine 50, a boom operation device which accepts an operation about a boom of the first selected work machine 40 or the second selected work machine 50, an arm operation device which accepts an operation about an arm of the first selected work machine 40 or the second selected work machine 50, and a bucket operation device which accepts an operation about an attachment such as a bucket of the first selected work machine 40 or the second selected work machine 50.
  • the remote operation mechanism 211 includes an operation device for another designation operation for performing the other designation operation.
  • another designation operation includes an action which possibly causes interference between an object present around the cooperating work machine and the cooperating work machine in a case where the cooperating work machine is caused to perform the operation, for example.
  • the operation device for another designation operation has an operation lever which accepts a rotation operation.
  • an operation lever (traveling lever) of the traveling operation device is operated to move a lower traveling body 410 of the work machine 40.
  • the traveling lever may serve also as a traveling pedal.
  • a traveling pedal may be provided which is fixed to a base portion or a lower end portion of the traveling lever.
  • An operation lever (revolution lever) of the revolution operation device is operated to move a hydraulic revolution motor which configures a revolution mechanism 430 of the work machine 40.
  • An operation lever (boom lever) of the boom operation device is operated to move a boom cylinder 442 of the work machine 40.
  • An operation lever (arm lever) of the arm operation device is operated to move an arm cylinder 444 of the work machine 40.
  • An operation lever (bucket lever) of the bucket operation device is operated to move a bucket cylinder 446 of the work machine 40.
  • the operation levers configuring the remote operation mechanism 211 are arranged around a seat St on which an operator OP1 is seated, for example.
  • the seat St is in a form of a high-back chair with armrests but may be a seated portion in an arbitrary form on which the operator OP1 can be seated such as a form of a low-back chair without a headrest or a form of a chair without a backrest.
  • a pair of left and right traveling levers 2110 corresponding to left and right continuous tracks are arranged laterally side by side in a left-right direction in front of the seat St.
  • One operation lever may also as plural operation levers.
  • a left operation lever 2111 provided in front of a left frame of the seat St which is illustrated in FIG. 3
  • a right operation lever 2112 provided in front of a right frame of the seat St, which is illustrated in FIG.
  • Lever patterns may arbitrarily be changed by an operation instruction from the operator OP1.
  • the hand terminal 2240 is a display device which is installed in obliquely right front when seen from a seated position of the seat St, for example.
  • the hand terminal 2240 has an input unit such as a touch panel, for example.
  • the hand terminal 2240 is capable of selecting the first selected work machine 40 or the second selected work machine 50 to be connected with the remote operation device 20 by the input unit such as a touch panel, for example.
  • a display screen and an input surface of the hand terminal 2240 is directed to the seat St side.
  • the hand terminal 2240 may be configured to be capable of angle adjustment of the display screen.
  • the display screen of the hand terminal 2240 displays, in its upper section 22401, images of plural work machines A to C as candidates for connection and displays, in its lower section 22402, the work machines A to C surrounded by outer frames.
  • the upper section 22401 displays a map which indicates the position relationship among the plural work machines A to C
  • the lower section 22402 displays respective machine numbers (actual machine identification information), specifications, and so forth of the work machines A to C.
  • a display form (such as a color of the outer frame, a thickness of frame lines of the outer frame, flashing of an icon of the work machine, or a background color) of the work machine A, B, or C selected by the tapping is changed in the lower section 22402 such that the selected work machine is identifiable against non-selected work machines.
  • the remote image output device 221 is configured with a central remote image output device 2210, a left remote image output device 2211, and a right remote image output device 2212 which have generally rectangular screens and are respectively arranged in front, in obliquely left front, and in obliquely right front of the seat St, for example.
  • the respective shapes and sizes of the screens (image display regions) of the central remote image output device 2210, the left remote image output device 2211, and the right remote image output device 2212 may be the same or different.
  • the hand terminal 2240 is capable of selecting the first selected work machine 40 or the second selected work machine 50 to be connected with the remote operation device 20 by the input unit such as the hand terminal 2240, for example.
  • the respective screens of the central remote image output device 2210, the left remote image output device 2211, and the right remote image output device 2212 may be parallel with a perpendicular direction or may be inclined with respect to the perpendicular direction.
  • At least one image output device among the central remote image output device 2210, the left remote image output device 2211, and the right remote image output device 2212 may be configured with an image output device which is divided into plural portions.
  • the central remote image output device 2210 may be configured with a pair of image output devices which have generally rectangular screens and are adjacent to each other in an up-down direction.
  • the remote sound output device 222 is configured with one or plural speakers and is, as illustrated in FIG. 3 , configured with a central remote sound output device 2220, a left remote sound output device 2221, and a right remote sound output device 2222 which are respectively arranged in the rear of the seat St, in a rear portion of the left armrest, and in a rear portion of the right armrest, for example. Respective specifications of the central remote sound output device 2220, the left remote sound output device 2221, and the right remote sound output device 2222 may be the same or different.
  • the first selected work machine 40 comprises an actual machine control device 400, an actual machine input interface 41, an actual machine output interface 42, and a positioning device 460.
  • the actual machine control device 400 is configured with an arithmetic processing unit (a single-core processor or a multi-core processor or a processor core configuring that), reads necessary data and software from a storage device such as a memory, and executes a computation process following the software for the data as a target.
  • arithmetic processing unit a single-core processor or a multi-core processor or a processor core configuring that
  • the positioning device 460 is a device which detects the position of the first selected work machine 40 and is configured with a global navigation satellite system (GNSS) receiver, for example.
  • GNSS global navigation satellite system
  • the first selected work machine 40 is a hydraulic, electric, or hybrid drive crawler excavator (construction machine), the hybrid drive being a combination of hydraulic and electric drives, for example, and comprises, as illustrated in FIG. 5 , the crawler lower traveling body 410 and an upper revolving body 420 which is revolvably mounted on the lower traveling body 410 via the revolution mechanism 430.
  • a cab 424 (operator cab) is provided to a front left side portion of the upper revolving body 420.
  • a work mechanism 440 is provided to a front central portion of the upper revolving body 420.
  • the actual machine input interface 41 comprises an actual machine operation mechanism 411, the actual machine image-capturing device 412, and an actual machine state sensor group 414.
  • the actual machine operation mechanism 411 comprises plural push buttons and plural operation levers, which are arranged similarly to the remote operation mechanism 211, around a seat arranged in an internal portion of the cab 424.
  • a drive mechanism or a robot which receives a signal corresponding to an operation manner of the plural push buttons and the plural operation levers and moves actual machine operation buttons and actual machine operation levers based on the received signal, is provided to the cab 424.
  • the actual machine image-capturing device 412 is installed in the internal portion of the cab 424, for example, and captures an image of an environment including at least a portion of the work mechanism 440 through a front window and a pair of left and right side windows. A portion or all of the front window (or a window frame) and the side windows may be omitted.
  • the actual machine image-capturing device 412 captures an image such that the image includes a sticker ST, in which the actual machine identification information (No. ⁇ in FIG. 6 ) corresponding to the actual machine identification information assigned to the first selected work machine 40 is indicated.
  • the sticker ST in which the actual machine identification information is indicated is pasted on the front window of the first selected work machine 40 or the second selected work machine 50, for example.
  • Images captured by the actual machine image-capturing device 412 are sequentially transmitted to the work support server 10.
  • the actual machine identification information is not limited to the sticker ST but may be engraved on a plate or may be engraved on a frame of the first selected work machine 40, for example.
  • the cab 424 is provided with a wiper device which wipes water drops adhered on the front window and a headlight device which is capable of projecting light forward.
  • the actual machine image-capturing device 412 is installed such that a changing situation, which changes in accordance with one designation operation of the wiper device, the headlight, or the like appears in captured images.
  • the actual machine state sensor group 414 is configured with angle sensors for respectively measuring a rotation angle (derricking angle) of a boom 441 with respect to the upper revolving body 410, a rotation angle of an arm 443 with respect to the boom 441, and a rotation angle of a bucket 445 with respect to the arm 443, a revolution angle sensor for measuring a revolution angle of the upper revolving body 420 with respect to the lower traveling body 410, an external force sensor for measuring an external force exerted on the bucket 445, a three-axis acceleration sensor for measuring three-axis acceleration exerted on the upper revolving body 420, and so forth.
  • the actual machine output interface 42 comprises an actual machine image output device 421 and an actual machine wireless communication apparatus 422.
  • the actual machine image output device 421 is arranged in the internal portion of the cab 424 and in the vicinity of the front window, for example.
  • the actual machine image output device 421 may be omitted.
  • the work mechanism 440 as a work mechanism comprises the boom 441 which is attached to the upper revolving body 420 to be capable of derricking, the arm 443 which is rotatably coupled with a distal end of the boom 441, and the bucket 445 which is rotatably coupled with a distal end of the arm 443.
  • the boom cylinder 442, the arm cylinder 444, and the bucket cylinder 446 which are configured with hydraulic cylinders capable of extending and contracting are attached.
  • various attachments such as a nibbler, a cutter, and a magnet may be used.
  • the boom cylinder 442 is interposed between the boom 441 and the upper revolving body 420 such that the boom cylinder 442 extends and contracts by being supplied with hydraulic oil so as to rotate the boom 441 in a derricking direction.
  • the arm cylinder 444 is interposed between the arm 443 and the boom 441 such that the arm cylinder 444 extends and contracts by being supplied with hydraulic oil so as to rotate the arm 443 around a horizontal axis with respect to the boom 441.
  • the bucket cylinder 446 is interposed between the bucket 445 and the arm 443 such that the bucket cylinder 446 extends and contracts by being supplied with hydraulic oil so as to rotate the bucket 445 around a horizontal axis with respect to the arm 443.
  • the second selected work machine 50 comprises an actual machine control device 500, an actual machine input interface 51, an actual machine output interface 52, and a positioning device 560.
  • the actual machine control device 500 is configured with an arithmetic processing unit (a single-core processor or a multi-core processor or a processor core configuring that), reads necessary data and software from a storage device such as a memory, and executes a computation process following the software for the data as a target.
  • arithmetic processing unit a single-core processor or a multi-core processor or a processor core configuring that
  • the positioning device 560 is a device which detects the position of the second selected work machine 50 and is configured with a global navigation satellite system (GNSS) receiver, for example.
  • GNSS global navigation satellite system
  • the position information of the second selected work machine 50, which is detected by the positioning device 560, is periodically transmitted to the work support server 10 and is stored in the database 102.
  • the second selected work machine 50 is a hydraulic, electric, or hybrid drive crawler excavator (construction machine), the hybrid drive being a combination of hydraulic and electric drives, for example, and comprises, as illustrated in FIG. 5 , a crawler lower traveling body 510 and an upper revolving body 520 which is revolvably mounted on the lower traveling body 510 via a revolution mechanism 530.
  • a cab 524 (operator cab) is provided to a front left side portion of the upper revolving body 520.
  • a work mechanism 540 is provided to a front central portion of the upper revolving body 520.
  • the actual machine input interface 51 comprises an actual machine operation mechanism 511, the actual machine image-capturing device 512, and an actual machine state sensor group 514.
  • the actual machine operation mechanism 511 comprises plural push buttons and plural operation levers, which are arranged similarly to the remote operation mechanism 211, around a seat arranged in an internal portion of the cab 524.
  • a drive mechanism or a robot which receives a signal corresponding to an operation manner of the plural push buttons and the plural operation levers and moves actual machine operation buttons and actual machine operation levers based on the received signal, is provided to the cab 524.
  • the actual machine image-capturing device 512 is installed in the internal portion of the cab 524, for example, and captures an image of an environment including at least a portion of the work mechanism 540 through a front window and a pair of left and right side windows. A portion or all of the front window (or a window frame) and the side windows may be omitted.
  • the actual machine image-capturing device 512 captures an image such that the image includes a sticker ST, in which the actual machine identification information assigned to the second selected work machine 50 is indicated.
  • the sticker ST in which the actual machine identification information is indicated is pasted on the front window of the second selected work machine 50, for example.
  • Images captured by the actual machine image-capturing device 512 are sequentially transmitted to the work support server 10 and is stored in the database 102.
  • the actual machine identification information is not limited to the sticker ST but may be engraved on a plate or may be engraved on a frame of the second selected work machine 50, for example.
  • the cab 524 is provided with a wiper device which wipes water drops adhered on the front window and a headlight device which is capable of projecting light forward.
  • the actual machine image-capturing device 512 is installed such that a changing situation, which changes in accordance with one designation operation of the wiper device, the headlight, or the like appears in captured images.
  • the actual machine state sensor group 514 is configured with angle sensors for respectively measuring a rotation angle (derricking angle) of a boom 541 with respect to the upper revolving body 520, a rotation angle of an arm 543 with respect to the boom 541, and a rotation angle of a bucket 545 with respect to the arm 543, a revolution angle sensor for measuring a revolution angle of the upper revolving body 520 with respect to the lower traveling body 510, an external force sensor for measuring an external force exerted on the bucket 545, a three-axis acceleration sensor for measuring three-axis acceleration exerted on the upper revolving body 520, and so forth.
  • the actual machine output interface 52 comprises an actual machine image output device 521 and an actual machine wireless communication apparatus 522.
  • the actual machine image output device 521 is arranged in the internal portion of the cab 524 and in the vicinity of the front window, for example.
  • the actual machine image output device 521 may be omitted.
  • the work mechanism 540 as a work mechanism comprises the boom 541 which is attached to the upper revolving body 520 to be capable of derricking, the arm 543 which is rotatably coupled with a distal end of the boom 541, and the bucket 545 which is rotatably coupled with a distal end of the arm 543.
  • a boom cylinder 542, an arm cylinder 544, and a bucket cylinder 546 which are configured with hydraulic cylinders capable of extending and contracting are attached.
  • various attachments such as a nibbler, a cutter, and a magnet may be used.
  • the boom cylinder 542 is interposed between the boom 541 and the upper revolving body 520 such that the boom cylinder 542 extends and contracts by being supplied with hydraulic oil so as to rotate the boom 541 in a derricking direction.
  • the arm cylinder 544 is interposed between the arm 543 and the boom 541 such that the arm cylinder 544 extends and contracts by being supplied with hydraulic oil so as to rotate the arm 543 around a horizontal axis with respect to the boom 541.
  • the bucket cylinder 546 is interposed between the bucket 545 and the arm 543 such that the bucket cylinder 546 extends and contracts by being supplied with hydraulic oil so as to rotate the bucket 545 around a horizontal axis with respect to the arm 543.
  • the display process starts with activation of the remote operation device 20 being a trigger.
  • the remote operation device 20 When the remote operation device 20 is activated by the operator OP1, the remote operation device 20 generates activation information as information that the remote operation device 20 has been activated and transmits the activation information to the work support server 10 and the work machines A to C (step S201).
  • Each of the work machines A to C starts a process with reception of the activation information being a trigger.
  • each of the work machines A to C periodically acquires position information for the work support server 10 (step S401) and transmits the acquired position information to the work support server 10 (step S402).
  • the work support server 10 starts a process with reception of the activation information being a trigger.
  • the work support server 10 acquires the position information transmitted from each of the work machines A to C by referring to the database 102 and generates a display image (Step S101).
  • the display image is generated based on the position information of each of the work machines A to C and the map data in step S101.
  • the map data is created based on map information, an aerial photograph, or the like of the work site.
  • the display image is generated by superimposing icons of the first selected work machine 40 and the second selected work machine 50 in respective positions corresponding to the pieces of position information on the map data, for example. It is desirable that a scale of the map data can appropriately be changed in accordance with a form of practice and that the position information of each of the work machines capable of being connected be included on any scale.
  • the work support server 10 transmits the display image generated in step S101 to the remote operation device 20 (step S102).
  • the remote operation device 20 When the display image is received, for example, as illustrated in FIG. 4 , the remote operation device 20 outputs the display image to the upper section 22401 of the display screen of the hand terminal 2240, thereby displays the display image (step S202), and finishes the process when the activation of the remote operation device 20 is finished, for example.
  • the display image is desirably displayed in the upper section 22401 of the display screen of the hand terminal 2240 such that the first selected work machine 40 or the second selected work machine 50 as a candidate to be connected with the remote operation device 20 is selectable.
  • the display image is displayed in a bird's eye manner, and the operator OP1 is enabled to recognize the positions of the plural work machines A to C as candidates to be connected in the work site.
  • the work support system starts the work support process with execution of the process in step S202 of the display process by the remote operation device 20 being a trigger.
  • the operator OP1 selects the first selected work machine 40 or the second selected work machine 50 as a candidate to be connected by touching an image of the work machine displayed on the hand terminal 2240, for example.
  • the remote operation device 20 determines whether or not selection of the work machine to be connected with the remote operation device 20 is performed, that is, a designation operation is performed (step S211).
  • the remote operation device 20 transmits information (selection information) of the work machine selected by the operator OP1 to the work support server 10 (step S212).
  • the first support process element 121 of the work support server 10 stores the selection information in the database 102.
  • the work support server 10 starts the work support process with reception of the selection information transmitted from the remote operation device 20 being a trigger.
  • the first support process element 121 of the work support server 10 refers to the database 102, chooses the work machine corresponding to the selection information based on the actual machine identification information, and requests an actual machine image-capturing device mounted on a first work machine (hereinafter, also referred to as a first communication work machine) to transmit an image through the first communication path identified by the first identifier of the chosen work machine (step Sill).
  • a first work machine hereinafter, also referred to as a first communication work machine
  • the actual machine image-capturing device of the first communication work machine starts a process with reception of a transmission request being a trigger.
  • the actual machine image-capturing device of the first communication work machine transmits a captured image to the work support server 10 through the first communication path (step S411).
  • the first support process element 121 of the work support server 10 acquires the captured image transmitted from the actual machine image-capturing device of the first communication work machine (step S112), transmits the captured image to the remote operation device 20, and causes the remote image output device 221 to output the captured image (step S113). Consequently, the first support process element 121 is capable of acquiring a captured image which is captured by the actual machine image-capturing device mounted on the first communication work machine and of causing the remote image output device 221 of the remote operation device 20 to output the captured image.
  • the remote operation device 20 sequentially outputs the received captured images to the remote image output device 221, for example. That is, the remote operation device 20 continues to output captured images transmitted from the work support server 10 to the remote image output device 221 until the process is finished.
  • the remote operation device 20 displays an indication such as a message which advises execution of one designation operation on the remote image output device 221, for example (step S213).
  • Examples of the message which advises execution of one designation operation may include “actuate the wipers", “light the headlights”, and so forth.
  • the remote operation device 20 transmits a command signal for executing one designation operation to the work support server 10 (step S214).
  • the second support process element 122 of the work support server 10 chooses the work machine corresponding to the selection information based on the actual machine identification information and transmits the command signal for executing one designation operation, which is transmitted from the remote operation device 20, to a second work machine (hereinafter, also referred to as a second communication work machine) via the second communication path identified by the second identifier of the chosen work machine (step S114).
  • a second work machine hereinafter, also referred to as a second communication work machine
  • the second support process element 122 of the work support server 10 is capable of recognizing that one designation operation is performed, which is a portion of plural kinds of operations for causing the second communication work machine among plural selected work machines to act, the second communication work machine establishing communication with the remote operation device 20, the communication being associated with the second communication path by the selection information, and in which a changing situation of the action corresponding to the operation appears in a captured image, and is capable of transmitting the command signal corresponding to the one designation operation to the second communication work machine.
  • the second communication work machine executes one designation operation (step S412).
  • the operator OP1 checks the captured image by the actual machine image-capturing device of the first communication work machine, which is displayed on the remote image output device 221 of the remote operation device 20, and can thereby check whether or not one designation operation is executed in the second communication work machine.
  • the operator OP1 cannot check the circumstance of one designation operation executed by the second communication work machine by the captured image by the actual machine image-capturing device of the first communication work machine.
  • the operator OP1 can check the circumstance of one designation operation executed by the second communication work machine by the captured image by the actual machine image-capturing device of the first communication work machine.
  • the operator OP1 inputs whether one designation operation is performed in the second communication work machine by touching the hand terminal 2240, for example.
  • the remote operation device 20 determines whether or not one designation operation is executed in the second communication work machine based on the input by the operator (step S215).
  • step S215 When it is determined that one designation operation is executed in the second communication work machine (step S215: Y), the remote operation device 20 transmits proper-state information indicating that the first communication work machine and the second communication work machine are the same selected work machine to the work support server 10 (step S216).
  • step S215: N When it is determined that one designation operation is not executed in the second communication work machine (step S215: N), the remote operation device 20 transmits improper-state information indicating that the first communication work machine and the second communication work machine are not the same selected work machine to the work support server 10 (step S217).
  • the second support process element 122 of the work support server 10 determines whether or not a cancellation condition for another designation operation is satisfied based on the proper-state information or the improper-state information which is transmitted from the remote operation device 20 (step S115).
  • the second support process element 122 of the work support server 10 starts acceptance of the other designation operation (step S116). That is, in the present embodiment, the second support process element 122 determines that a cancellation operation is performed in the remote operation device based on reception of the proper-state information. Further, after acceptance of the other designation operation is started in step S116, the second support process element 122 transmits the command signal corresponding to the other designation operation, which is transmitted from the remote operation device 20, to the second communication work machine. Accordingly, based on a remote operation command corresponding to an operation manner of the remote operation mechanism 211, the operation manner being recognized by the remote control device 200 of the remote operation device 20, actions by the work mechanism and so forth of the second communication work machine are controlled.
  • step S115: N When it is determined that the cancellation condition for the other designation operation is not satisfied (step S115: N), the second support process element 122 of the work support server 10 maintains suspension of acceptance of the other designation operation (step S117) and finishes the work support process.
  • the operator OP1 can visually check a captured image in which one designation operation is executed and which is displayed on the remote image output device 221. Accordingly, the operator OP1 is enabled to check whether the first selected work machine 40 or the second selected work machine 50 with which the remote operation device 20 is connected agrees with the selection information and the first communication work machine and second communication work machine.
  • the captured image which is captured by the actual machine image-capturing device mounted on the first communication work machine is acquired via the first communication path based on the selection information specifying the cooperating work machine, and the command signal corresponding to one designation operation is transmitted to the second communication work machine. Consequently, through the captured image, the operator can check whether or not the first communication work machine acts in accordance with one designation operation performed for the second communication work machine, that is, whether or not an operation performed for the cooperating work machine as the remote operation target is performed in the cooperating work machine. Accordingly, the operator can check a connection situation with the work machine as the remote operation target.
  • the actual machine identification information is together displayed in the display image, and the operator OP1 can thereby visually check whether the actual machine identification information of the captured image displayed in the remote image output device 221 matches the actual machine identification information in the display image and is thus enabled to quickly and accurately perform checking work.
  • the work support process is executed by the work support server 10.
  • the work support process may be performed by the remote operation device 20 or may be performed by the first selected work machine 40 or the second selected work machine 50.
  • an image is acquired which is captured by the actual machine image-capturing device 412 of the first selected work machine 40 or the actual machine image-capturing device 512 of the second selected work machine 50.
  • an image may be acquired which is captured by another image-capturing device than the actual machine image-capturing device 412 of the first selected work machine 40 or the actual machine image-capturing device 512 of the second selected work machine 50.
  • Examples of such an image-capturing device may include a fixed point image-capturing device provided in a position where an image of an external appearance of the first selected work machine 40 or the second selected work machine 50 is capable of being captured and an image-capturing device provided to an aerial vehicle such as a drone.
  • the sticker ST in which the actual machine identification information is indicated is preferably provided in a position which is visually recognizable from the outside of the first selected work machine 40 and the second selected work machine 50.
  • Examples of such a position may include side surfaces of the upper revolving body 420 of the first selected work machine 40 and of the upper revolving body 520 of the second selected work machine 50.
  • the captured image which is captured by the actual machine image-capturing device mounted on the first work machine is acquired via the first communication path based on the selection information specifying the cooperating work machine, and the command signal corresponding to one designation operation is transmitted to the second work machine. Consequently, through the captured image, the operator can check whether or not the first work machine acts in accordance with one designation operation performed for the second work machine, that is, whether or not an operation performed for the cooperating work machine as the remote operation target is performed in the cooperating work machine. Accordingly, the operator can check the connection situation with the work machine as the remote operation target.
  • the second support process element preferably suspends transmission of a command signal corresponding to another designation operation for causing the cooperating work machine to act to the second work machine in a case where a cancellation condition is not satisfied and preferably permits transmission of the command signal corresponding to the other designation operation for causing the cooperating work machine to act to the second work machine in a case where the cancellation condition is satisfied, and the cancellation condition preferably includes a fact that a cancellation operation is performed in the remote operation device.
  • the work machine can be prevented from running due to another designation operation. After the operator checks the connection situation, the work machine can remotely be operated by operations including the other designation operation.
  • the second support process element preferably acquires a determination result about whether or not the one designation operation is executed, preferably permits another designation operation for causing the cooperating work machine to act in a case where the one designation operation is executed, and preferably transmits a command signal corresponding to the other designation operation to the second work machine.
  • the second support process element preferably suspends transmission of a command signal corresponding to another designation operation for causing the cooperating work machine to act to the second work machine in a case where a cancellation condition is not satisfied and preferably permits transmission of the command signal corresponding to the other designation operation for causing the cooperating work machine to act to the second work machine in a case where the cancellation condition is satisfied, and the cancellation condition preferably includes a fact that in a case where the one designation operation is executed, the second support process element acquires a determination result about whether or not a changing situation of an action corresponding to the one designation operation is present in the captured image, and the changing situation of the action corresponding to the one designation operation is present.
  • connection situation with the work machine as the remote operation target can automatically be assessed, checking work by the operator can be reduced.
  • the first support process element preferably acquires position data of the plural work machines as the remote operation target, and the second support process element preferably generates a display image in which each of the plural work machines appears based on the plural sets of position data and preferably transmits the display image to the remote operation device.
  • the operator can check a circumstance where each of the plural work machines as the remote operation target appears in the display image and can thus recognize relative positions of the work machines in the work site.

Landscapes

  • Engineering & Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Mining & Mineral Resources (AREA)
  • Civil Engineering (AREA)
  • Structural Engineering (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Remote Sensing (AREA)
  • General Physics & Mathematics (AREA)
  • Physics & Mathematics (AREA)
  • Automation & Control Theory (AREA)
  • Aviation & Aerospace Engineering (AREA)
  • Signal Processing (AREA)
  • General Health & Medical Sciences (AREA)
  • Multimedia (AREA)
  • Health & Medical Sciences (AREA)
  • Computing Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Medical Informatics (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Operation Control Of Excavators (AREA)
  • Control Of Position, Course, Altitude, Or Attitude Of Moving Bodies (AREA)
  • Selective Calling Equipment (AREA)
  • Component Parts Of Construction Machinery (AREA)
EP22160221.2A 2021-03-29 2022-03-04 Système de contrôle à distance Active EP4079977B1 (fr)

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
JP2021055067A JP2022152338A (ja) 2021-03-29 2021-03-29 作業支援システムおよび作業支援複合システム

Publications (2)

Publication Number Publication Date
EP4079977A1 true EP4079977A1 (fr) 2022-10-26
EP4079977B1 EP4079977B1 (fr) 2024-07-24

Family

ID=80937255

Family Applications (1)

Application Number Title Priority Date Filing Date
EP22160221.2A Active EP4079977B1 (fr) 2021-03-29 2022-03-04 Système de contrôle à distance

Country Status (4)

Country Link
US (1) US20220307224A1 (fr)
EP (1) EP4079977B1 (fr)
JP (1) JP2022152338A (fr)
CN (1) CN115150441A (fr)

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2019176401A (ja) 2018-03-29 2019-10-10 コベルコ建機株式会社 作業機械の遠隔操作システム
WO2020250558A1 (fr) * 2019-06-10 2020-12-17 コベルコ建機株式会社 Système de manœuvre à distance et serveur de manœuvre à distance
WO2021019948A1 (fr) * 2019-07-29 2021-02-04 コベルコ建機株式会社 Système d'identification de position pour engin de chantier

Family Cites Families (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP3217723B2 (ja) * 1997-03-13 2001-10-15 ▲すすむ▼ 舘 遠隔通信システム及び遠隔通信方法
CN102282321B (zh) * 2009-01-20 2013-08-28 胡斯华纳有限公司 用于遥控工作机的控制系统
CN104205792B (zh) * 2012-09-28 2016-08-24 株式会社久保田 作业机的数据通信系统
US20140170617A1 (en) * 2012-12-19 2014-06-19 Caterpillar Inc. Monitoring System for a Machine
WO2014177755A1 (fr) * 2013-04-30 2014-11-06 Tana Oy Commande de machine de travail
AU2016242761B2 (en) * 2015-03-31 2019-08-08 Komatsu Ltd. Working machine
US10794047B2 (en) * 2015-07-15 2020-10-06 Komatsu Ltd. Display system and construction machine
WO2016047806A1 (fr) * 2015-09-30 2016-03-31 株式会社小松製作所 Dispositif de surveillance de la périphérie pour machine-outil sur chenilles
US10795351B2 (en) * 2016-07-19 2020-10-06 Raven Industries, Inc. System and method for autonomous control of agricultural machinery and equipment
JP6889389B2 (ja) * 2016-09-15 2021-06-18 コベルコ建機株式会社 稼働管理システム及び稼働情報の開示方法
JP6922806B2 (ja) * 2018-03-23 2021-08-18 コベルコ建機株式会社 遠隔操作システム及び主操作装置
JP6947101B2 (ja) * 2018-03-28 2021-10-13 コベルコ建機株式会社 遠隔操作システム及び主操作装置

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2019176401A (ja) 2018-03-29 2019-10-10 コベルコ建機株式会社 作業機械の遠隔操作システム
EP3745716A1 (fr) * 2018-03-29 2020-12-02 Kobelco Construction Machinery Co., Ltd. Système de fonctionnement à distance destiné à une machine de travail
WO2020250558A1 (fr) * 2019-06-10 2020-12-17 コベルコ建機株式会社 Système de manœuvre à distance et serveur de manœuvre à distance
EP3951094A1 (fr) * 2019-06-10 2022-02-09 Kobelco Construction Machinery Co., Ltd. Système de manoeuvre à distance et serveur de manoeuvre à distance
WO2021019948A1 (fr) * 2019-07-29 2021-02-04 コベルコ建機株式会社 Système d'identification de position pour engin de chantier
EP3985182A1 (fr) * 2019-07-29 2022-04-20 Kobelco Construction Machinery Co., Ltd. Système d'identification de position pour engin de chantier

Also Published As

Publication number Publication date
CN115150441A (zh) 2022-10-04
US20220307224A1 (en) 2022-09-29
JP2022152338A (ja) 2022-10-12
EP4079977B1 (fr) 2024-07-24

Similar Documents

Publication Publication Date Title
EP4043646B1 (fr) Serveur d'aide au fonctionnement à distance, procédé d'aide au fonctionnement à distance et système d'aide au fonctionnement à distance
AU2017242392B2 (en) Terminal device, control device, data-integrating device, work vehicle, image-capturing system, and image-capturing method
EP4265857A1 (fr) Système d'aide au travail et système composite d'assistance au travail
WO2021124654A1 (fr) Serveur d'aide au chantier, procédé d'aide au chantier, et système d'aide au chantier
EP4072130A1 (fr) Système d'assistance au travail et procédé d'assistance au travail
EP4130400A1 (fr) Dispositif d'aide au fonctionnement à distance et procédé d'aide au fonctionnement à distance
EP4079977A1 (fr) Système de contrôle à distance
EP4283051A1 (fr) Système d'aide au travail et système composite d'aide au travail
US12123175B2 (en) Remote operation device, remote operation assistance server, remote operation assistance system, and remote operation assistance method
WO2021106278A1 (fr) Serveur d'aide au travail, procédé d'aide au travail et système d'aide au travail
WO2021106281A1 (fr) Dispositif d'aide au travail, procédé d'aide au travail et système d'aide au travail
WO2021106279A1 (fr) Serveur d'aide au travail, procédé d'aide au travail et système d'aide au travail
JP7528892B2 (ja) 作業支援システムおよび作業支援複合システム
WO2021106280A1 (fr) Serveur d'aide au travail, procédé d'aide au travail, et système d'aide au travail
JP2023103876A (ja) 遠隔操作支援システム

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE APPLICATION HAS BEEN PUBLISHED

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20230420

RBV Designated contracting states (corrected)

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

INTG Intention to grant announced

Effective date: 20240229

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

Ref country code: DE

Ref legal event code: R096

Ref document number: 602022004702

Country of ref document: DE