CN113614317B - Remote operation system and remote operation server - Google Patents
Remote operation system and remote operation server Download PDFInfo
- Publication number
- CN113614317B CN113614317B CN201980094527.6A CN201980094527A CN113614317B CN 113614317 B CN113614317 B CN 113614317B CN 201980094527 A CN201980094527 A CN 201980094527A CN 113614317 B CN113614317 B CN 113614317B
- Authority
- CN
- China
- Prior art keywords
- remote operation
- notification
- work machine
- operation device
- server
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000007246 mechanism Effects 0.000 claims description 23
- 238000003384 imaging method Methods 0.000 claims description 10
- 239000000470 constituent Substances 0.000 claims description 3
- 238000004891 communication Methods 0.000 description 26
- 230000006870 function Effects 0.000 description 13
- 238000001514 detection method Methods 0.000 description 7
- 238000000034 method Methods 0.000 description 7
- 238000010276 construction Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 6
- 238000009412 basement excavation Methods 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 230000000007 visual effect Effects 0.000 description 4
- 230000005540 biological transmission Effects 0.000 description 3
- 239000010720 hydraulic oil Substances 0.000 description 3
- 230000036772 blood pressure Effects 0.000 description 2
- 210000004556 brain Anatomy 0.000 description 2
- 230000007613 environmental effect Effects 0.000 description 2
- 230000035790 physiological processes and functions Effects 0.000 description 2
- 239000002689 soil Substances 0.000 description 2
- 230000009471 action Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000003825 pressing Methods 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
Images
Classifications
-
- E—FIXED CONSTRUCTIONS
- E02—HYDRAULIC ENGINEERING; FOUNDATIONS; SOIL SHIFTING
- E02F—DREDGING; SOIL-SHIFTING
- E02F9/00—Component parts of dredgers or soil-shifting machines, not restricted to one of the kinds covered by groups E02F3/00 - E02F7/00
- E02F9/26—Indicating devices
- E02F9/261—Surveying the work-site to be treated
-
- E—FIXED CONSTRUCTIONS
- E02—HYDRAULIC ENGINEERING; FOUNDATIONS; SOIL SHIFTING
- E02F—DREDGING; SOIL-SHIFTING
- E02F9/00—Component parts of dredgers or soil-shifting machines, not restricted to one of the kinds covered by groups E02F3/00 - E02F7/00
- E02F9/20—Drives; Control devices
- E02F9/2025—Particular purposes of control systems not otherwise provided for
- E02F9/205—Remotely operated machines, e.g. unmanned vehicles
-
- E—FIXED CONSTRUCTIONS
- E02—HYDRAULIC ENGINEERING; FOUNDATIONS; SOIL SHIFTING
- E02F—DREDGING; SOIL-SHIFTING
- E02F9/00—Component parts of dredgers or soil-shifting machines, not restricted to one of the kinds covered by groups E02F3/00 - E02F7/00
- E02F9/20—Drives; Control devices
- E02F9/2025—Particular purposes of control systems not otherwise provided for
- E02F9/2054—Fleet management
-
- E—FIXED CONSTRUCTIONS
- E02—HYDRAULIC ENGINEERING; FOUNDATIONS; SOIL SHIFTING
- E02F—DREDGING; SOIL-SHIFTING
- E02F9/00—Component parts of dredgers or soil-shifting machines, not restricted to one of the kinds covered by groups E02F3/00 - E02F7/00
- E02F9/26—Indicating devices
- E02F9/261—Surveying the work-site to be treated
- E02F9/262—Surveying the work-site to be treated with follow-up actions to control the work tool, e.g. controller
-
- G—PHYSICS
- G05—CONTROLLING; REGULATING
- G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
- G05D1/0011—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots associated with a remote control arrangement
- G05D1/0027—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots associated with a remote control arrangement involving a plurality of vehicles, e.g. fleet or convoy travelling
-
- G—PHYSICS
- G05—CONTROLLING; REGULATING
- G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
- G05D1/0011—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots associated with a remote control arrangement
- G05D1/0038—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots associated with a remote control arrangement by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation
-
- G—PHYSICS
- G05—CONTROLLING; REGULATING
- G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
- G05D1/20—Control system inputs
- G05D1/22—Command input arrangements
- G05D1/221—Remote-control arrangements
- G05D1/222—Remote-control arrangements operated by humans
- G05D1/224—Output arrangements on the remote controller, e.g. displays, haptics or speakers
-
- G—PHYSICS
- G05—CONTROLLING; REGULATING
- G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
- G05D1/60—Intended control result
- G05D1/69—Coordinated control of the position or course of two or more vehicles
-
- G—PHYSICS
- G08—SIGNALLING
- G08C—TRANSMISSION SYSTEMS FOR MEASURED VALUES, CONTROL OR SIMILAR SIGNALS
- G08C17/00—Arrangements for transmitting signals characterised by the use of a wireless electrical link
- G08C17/02—Arrangements for transmitting signals characterised by the use of a wireless electrical link using a radio link
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04Q—SELECTING
- H04Q9/00—Arrangements in telecontrol or telemetry systems for selectively calling a substation from a main station, in which substation desired apparatus is selected for applying a control signal thereto or for obtaining measured values therefrom
-
- G—PHYSICS
- G08—SIGNALLING
- G08C—TRANSMISSION SYSTEMS FOR MEASURED VALUES, CONTROL OR SIMILAR SIGNALS
- G08C2201/00—Transmission systems of control signals via wireless link
- G08C2201/40—Remote control systems using repeaters, converters, gateways
- G08C2201/42—Transmitting or receiving remote control signals via a network
Landscapes
- Engineering & Computer Science (AREA)
- Mining & Mineral Resources (AREA)
- Civil Engineering (AREA)
- General Engineering & Computer Science (AREA)
- Structural Engineering (AREA)
- Radar, Positioning & Navigation (AREA)
- Remote Sensing (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Aviation & Aerospace Engineering (AREA)
- Automation & Control Theory (AREA)
- Computer Networks & Wireless Communication (AREA)
- Selective Calling Equipment (AREA)
- Telephonic Communication Services (AREA)
- Operation Control Of Excavators (AREA)
- Component Parts Of Construction Machinery (AREA)
Abstract
The invention provides a system which can avoid the instability of the operation state of a working machine when switching a remote operation main body of the working machine. When a work machine (40) is remotely operated by a 1 st remote operation device (10), if the state of one or both of the work machine (40) and a 1 st operator is a specified state, a 1 st notification corresponding to the specified state is transmitted to a remote operation server (30). In the remote operation server (30), an appropriate 2 nd remote operation device (20) is selected as a remote operation body of the working machine (40) in consideration of the content of the 1 st notification.
Description
Technical Field
The present invention relates to a system for remotely operating a working machine or the like.
Background
A technique for remotely operating a work machine has been proposed (for example, see patent document 1).
Documents of the prior art
Patent document
Patent document 1: japanese patent laid-open publication No. 2016-076801
Disclosure of Invention
Technical problems to be solved by the invention
However, when the remote operation body of the work machine is switched from one operator who operates one remote operation device to another operator who operates another remote operation device, the remote operation method of the other operator may become inappropriate in view of the operation state of the work machine corresponding to the remote operation method of the one operator, and the operation state of the work machine may become unstable.
Therefore, an object of the present invention is to provide a system that can avoid instability of the operating state of a work machine when switching a remote operation body of the work machine.
Solution for solving the above technical problem
A remote operation system of the present invention includes: a working machine having an imaging device for imaging a working mechanism and an environment including at least a part of the working mechanism; a plurality of remote operation devices having an operation mechanism for remotely operating the work machine and an image output device that displays a captured image captured by the imaging device of the work machine; and a remote operation server having a function of communicating with each of the plurality of remote operation devices.
In the remote operation system according to the present invention, a 1 st remote operation device among the plurality of remote operation devices includes: a state detector that detects a state of at least one of the work machine and the operator as a specified state; a 1 st client control device that, when the state detector detects that the state of at least one of the work machine and the operator is the designated state, transmits a 1 st notification corresponding to the designated state, the remote operation server including: a 1 st server arithmetic processing element that receives the 1 st notification and selects a 2 nd remote operation device different from the 1 st remote operation device among the plurality of remote operation devices based on the 1 st notification; and a 2 nd server arithmetic processing element that transmits a 2 nd notification corresponding to the 1 st notification to the 2 nd remote operation device or a 2 nd client control device included in the 2 nd remote operation device, and executes processing for changing a remote operation authority of the work machine from the 1 st remote operation device to the 2 nd remote operation device.
Further, a remote operation server according to the present invention has a function of communicating with each of a plurality of remote operation devices for remotely operating a work machine, and includes: a 1 st server arithmetic processing element that receives, by a 1 st remote operation device among the plurality of remote operation devices, a 1 st notification corresponding to a specified state transmitted from the 1 st remote operation device in response to detection that the state of at least one of the work machine and the operator is the specified state, and selects a 2 nd remote operation device, which is different from the 1 st remote operation device, among the plurality of remote operation devices based on the 1 st notification; and a 2 nd server arithmetic processing element that transmits a 2 nd notification corresponding to the 1 st notification to the 2 nd remote operation device, and executes processing for changing a remote operation authority of the work machine from the 1 st remote operation device to the 2 nd remote operation device.
Drawings
Fig. 1 is an explanatory diagram of a configuration of a remote operation system according to an embodiment of the present invention.
Fig. 2 is an explanatory diagram of the structure of the working machine.
Fig. 3 is an explanatory diagram of a configuration of the 1 st remote operation device.
Fig. 4 is an explanatory diagram of the 1 st function of the remote operation system according to the embodiment of the present invention.
Fig. 5 is an explanatory diagram of the 2 nd function of the remote operation system according to the embodiment of the present invention.
Fig. 6 is an explanatory diagram of an environment image.
Detailed Description
(constitution)
The remote operation system shown in fig. 1 as an embodiment of the present invention includes a 1 st remote operation device 10, a 2 nd remote operation device 20, a remote operation server 30, and a work machine 40. The remote operation server 30 can switch the remote operation body of the common work machine 40 between the 1 st remote operation device 10 and the 2 nd remote operation device 20.
(construction of work machine)
The work machine 40 includes a work machine control device 400, an imaging device 401, a wireless communication device 402, and an operating mechanism 440. The work machine control device 400 is configured by an arithmetic processing device (a single-core processor or a multi-core processor, or a processor core constituting the processor), reads necessary data and software from a storage device such as a memory, and executes arithmetic processing by the software with the data as an object.
The working machine 40 is, for example, a crawler excavator (construction machine), and as shown in fig. 2, includes a crawler-type lower traveling structure 410 and an upper revolving structure 420 that is rotatably mounted on the lower traveling structure 410 via a revolving mechanism 430. A cab (driver's cab) 422 is provided in a front left portion of the upper revolving structure 420. A work attachment 440 is provided in a front center portion of the upper revolving structure 420.
The work attachment 440 as the work mechanism includes: a boom 441 liftably mounted on the upper revolving body 420; an arm 443 rotatably coupled to the distal end of boom 441; bucket 445 is rotatably coupled to the tip of arm 443. A boom cylinder 442, an arm cylinder 444, and a bucket cylinder 446, which are telescopic hydraulic cylinders, are attached to the work attachment 440.
The imaging device 401 is provided inside the cab 422, for example, and images an environment including at least a part of the operating mechanism 440 through a front window of the cab 422.
The cab 422 is provided with: an actual machine operation lever corresponding to an operation lever (described later) constituting the 1 st remote operation device 10; the drive mechanism or the robot receives a signal corresponding to the operation mode of each operation lever from the remote control room, and operates the real machine operation lever based on the received signal.
(constitution of the 1 st remote control device)
The 1 st remote operation device 10 includes a 1 st client control device 100, a 1 st input interface 110, and a 1 st output interface 120. The 1 st client control device 100 is configured by an arithmetic processing device (a single-core processor or a multi-core processor, or a processor core constituting the processor), reads necessary data and software from a storage device such as a memory, and executes arithmetic processing by the software with the data as an object. The 1 st input interface 110 includes a 1 st operation mechanism 111 and a state detector 112. The 1 st output interface 120 includes a 1 st image output device 121 and a 1 st wireless communication device 122.
At least a part of the constituent elements of the 1 st remote operation device 10 (for example, constituent elements excluding the 1 st operation mechanism 111) may be constituted by a terminal 25 that can be carried by an operator, such as a smartphone or a tablet terminal. In this case, the place where the terminal 25 is placed may be set in the 1 st remote operation device 10.
The 1 st operating mechanism 111 includes a travel operating device, a swing operating device, a boom operating device, an arm operating device, and a bucket operating device. Each operating device has an operating lever that receives a rotational operation. An operation lever (travel lever) of the travel operation device is operated to operate the lower traveling structure 410. The travel bar may also double as a travel pedal. For example, a travel pedal fixed to the base or lower end of the travel lever may be provided. An operation lever (turning lever) of the turning operation device is operated to operate a hydraulic turning motor constituting the turning mechanism 430. An operation lever (boom lever) of the boom operation device is operated to operate the boom cylinder 442. The operation lever (arm m lever) of the arm operation device is operated to operate the arm cylinder 444. An operation lever (bucket lever) of the bucket operation device is operated to operate the bucket cylinder 446.
For example, as shown in fig. 3, the respective operation levers constituting the 1 st operation mechanism 111 are arranged around a seat 1100 on which an operator sits. The seat 1100 may be a high back seat with an armrest, a low back seat without a headrest, or a chair without a back, which is any type that an operator can sit on.
A pair of left and right travel bars 1110 corresponding to the left and right crawler belts are arranged in parallel in the left and right in front of the seat 1100. One operation lever can double as a plurality of operation levers. For example, the right operation lever 1111 provided in front of the right frame of the seat 1100 shown in fig. 3 functions as a swing arm lever when operated in the front-rear direction, and functions as a bucket lever when operated in the left-right direction. Similarly, the arm lever functions when the left operation lever 1112 provided in front of the left frame of the seat 1100 shown in fig. 3 is operated in the front-rear direction, and functions as a swing lever when the left operation lever 1112 is operated in the left-right direction. The lever mode can be arbitrarily changed in accordance with an operation instruction of the operator.
For example, as shown in fig. 3, the 1 st image output device 121 is composed of a right oblique front image output device 1211, a front image output device 1212, and a left oblique front image output device 1213 that are respectively disposed in the right oblique front, the front, and the left oblique front of the seat 1100. The image output devices 1211 to 1213 may further include a speaker (voice output device).
The state detector 112 detects that the state of one or both of the 1 st operator of the work machine 40 and the 1 st remote operation device 10 is a specified state. For example, a state in which a help button constituting the input interface 110 is pressed is detected as the specified state. The 1 st operator can press the help button when the operator feels uneasy about the remote operation of the work machine 40 due to the environment image displayed on the 1 st image output device 121 or the image indicating the operation state of the work machine 40.
The help button may be pressed by a physical button provided in the 1 st remote operation device 10 and capable of being pressed, or a virtual button may be displayed on a touch panel such as the terminal device 25 and the pressing of the virtual button position may be detected.
The designated state may be detected based on information on not only the help button but also the lack of skill of the 1 st operator for the operation, a reduction in work efficiency due to fatigue of the 1 st operator, physical discomfort, and the like.
At least one of the movement pattern of the line of sight of the 1 st operator, the sound emission pattern, the movement pattern of the predetermined body part such as the finger, the hand, the arm, or the shoulder, the brain wave, the perspiration amount, the heart rate, and the physiological information such as the blood pressure may be detected, and based on the detection result, the case where the psychological state or the physiological state of the 1 st operator estimated according to the predetermined algorithm is the designated psychological state or the designated physiological state may be detected as the designated state.
In this case, for example, the 1 st operator's operation is detected to be deviated from a predetermined operation such as an excavation operation, and the specified state can be detected based on the operation mode. Specifically, it is possible to detect whether or not the visual line direction detected by the visual line detection sensor for detecting the movement of the visual line of the 1 st operator deviates from the visual line direction assumed by the predetermined operation, or whether or not the operation of the operation lever or the operation pedal deviates from the operation assumed by the predetermined operation, and it is possible to detect the designated state according to the presence or absence of the deviation and the frequency of the deviation.
The action pattern may also be detected by the utterance pattern of the 1 st operator. For example, when the work machine 40 is equipped with a speaker for transmitting the voice of the 1 st operator to the surrounding operator, it is possible to detect a situation in which the surrounding operator is not presented with a voice at an appropriate timing or a main communicator such as a site supervisor is not contacted at an appropriate timing, and to detect a specified state based on the appropriateness of the voice manner.
The specified state can also be detected from the fact that the values of the brain waves, the amount of perspiration, the heart rate, the blood pressure, and the like deviate from normal values.
The operation state of the work machine 40 detected by the sensor provided in the work machine 40 and received by the 1 st wireless communication device 122 may be detected as the designated operation state. For example, a case where the inclination angle or inclination angular velocity of lower traveling structure 410 or upper revolving structure 420 with respect to the horizontal plane, which is detected by an inclination sensor provided in work machine 40, is equal to or greater than a threshold value may be detected as the predetermined state.
In this case, for example, in an operation performed by an operator with poor operation skills in remote operation, the inclination angle and the inclination angular velocity tend to increase frequently to be equal to or higher than the threshold value, and the specified state can be detected by detecting the state.
The case where the deviation of the work condition is detected by the work machine 40 remotely operated by the 1 st operator may be regarded as the specified state with reference to the construction plan. For example, based on the construction plan stored in the storage device constituting the 1 st client control device 100, the following may be detected as the designated state: that is, in a time zone in which a predetermined work is performed in a certain area, a state in which the predetermined work is performed in an area different from the certain area by the work machine 40 or a state in which another work (for example, an excavation work before a closing work) different from the predetermined work (for example, a closing work after excavation of the predetermined area is finished) is performed in the certain area by the positioning device configured by the GPS or the like and the imaging device 401 is detected.
(constitution of No. 2 remote control device)
The 2 nd remote operation device 20 includes a 2 nd client control device 200, a 2 nd input interface 210, and a 2 nd output interface 220. The 2 nd client control apparatus 200 is configured by an arithmetic processing apparatus (a single-core processor or a multi-core processor, or a processor core constituting the processor), reads necessary data and software from a storage apparatus such as a memory, and executes arithmetic processing by the software with the data as an object. The 2 nd input interface 210 includes a 2 nd operation mechanism 211. The 2 nd output interface 220 includes a 2 nd image output device 221 and a 2 nd wireless communication device 222.
The detailed configuration of the 2 nd remote operation device 20 is almost the same as that of the 1 st remote operation device 10 except that the functions of the status detector 112 and the 2 nd client control device 200 described later are omitted, and therefore, the description thereof is omitted (see fig. 3).
(constitution of remote operation Server)
The remote operation server 30 includes a 1 st server arithmetic processing element 31 and a 2 nd server arithmetic processing element 32. The 1 st server arithmetic processing element 31 receives the 1 st notification, and selects the 2 nd remote operation device 20 different from the 1 st remote operation device 10 among the plurality of remote operation devices based on the 1 st notification. The 2 nd server arithmetic processing element 32 transmits a 2 nd notification corresponding to the 1 st notification to the 2 nd remote operation device 20, and executes processing for changing the remote operation authority of the work machine 40 from the 1 st remote operation device 10 to the 2 nd remote operation device 20.
(function)
In the 1 st remote operation device 10, the 1 st operator performs a predetermined operation (fig. 4/step 102). The predetermined operation is, for example, an operation of a button or an operation lever constituting the 1 st input interface 110 or the 1 st operation mechanism 111. Accordingly, the 1 st client control device 100 transmits an operation start instruction to the work machine 40 from the 1 st remote operation device 10 via the remote operation server 30 via the 1 st wireless communication device 122 (fig. 4/step 104).
In the work machine 40, an operation start instruction is received from the remote operation server 30 via the wireless communication device 402 by using the work machine control device 400 (fig. 4/step 402). Accordingly, the work machine control device 400 outputs an instruction to the imaging device 401, and the imaging device 401 acquires the captured image according to the instruction (fig. 4/step 404). Captured image data representing the captured image is transmitted to the 1 st remote operation device 10 via the remote operation server 30 by the work machine control device 400 via the wireless communication device 402 (fig. 4/step 406).
In the 1 st remote operation device 10, the 1 st client control device 100 receives the captured image data from the remote operation server 30 via the 1 st wireless communication device 122 (fig. 4/step 106). The 1 st client control apparatus 100 displays an environment image (all or a part of the captured image itself, or a simulated environment image generated based on the captured image) corresponding to the captured image data on the 1 st image output apparatus 121 (fig. 4/step 108). As a result, for example, as shown in fig. 6, an environmental image including a boom 441, an arm 443, a bucket 445, and an arm cylinder 444, which are parts of the work attachment 440 as the work mechanism, is displayed on the 1 st image output device 121.
In the 1 st remote operation device 10, the 1 st operator operates the 1 st operation means 111 (fig. 4/step 110), and accordingly, the 1 st client control device 100 transmits an operation command corresponding to the operation mode to the work machine 40 via the remote operation server 30 via the 1 st wireless communication device 122 (fig. 4/step 112).
In work machine 40, an operation command is received from remote operation server 30 via wireless communication device 402 by work machine control device 400 (fig. 4/step 408). Accordingly, the work attachment 440 and the like are controlled in operation by the work machine control device 400 (fig. 4/step 410). For example, the work of scooping up the soil in front of the work machine 40 with the bucket 445, turning the upper revolving structure 420, and then dropping the soil from the bucket 445 is performed.
In the 1 st remote operation device 10, the state detector 112 determines whether or not the 1 st operator and/or the designated state of the work machine 40 is detected (fig. 5/step 114). If the determination result is negative (no in fig. 5/step 114), reception of the captured image data and output of the environmental image, and operation of the 1 st operating mechanism 111 and transmission of the operation command corresponding thereto are repeated (see fig. 4/steps 106, 108, 110, and 112).
On the other hand, if the determination result is affirmative (y in fig. 5/step 114), the 1 st notification corresponding to the state detection result is transmitted to the remote operation server 30 through the 1 st wireless communication device 122 by the 1 st client control device 100 (fig. 5/step 116). The "1 st notification" includes an environment image indicating the environment of the work machine 40 immediately before the detection of the specified state, and information necessary for remotely operating the work machine 40, such as voice information, a construction plan, and an operation state and a type of the work machine 40, if necessary.
In the remote operation server 30, the 1 st notification is received by the 1 st server arithmetic processing element 31 through the wireless communication device (fig. 5/step 302).
The 2 nd remote operation device 20 other than the 1 st remote operation device 10 among the plurality of remote operation devices is selected by the 1 st server arithmetic processing element 31 based on the 1 st notification (fig. 5/step 304). For example, for each operator, the following information is registered to the database: that is, whether the remote operation is in the middle of the remote operation, the time from the end of the remote operation in the case of the remote operation, the number of times of experience of the remote operation, the number of times of taking over the remote operation from the other operator halfway, the number of times of taking over by the other operator halfway in the remote operation, the state of the work machine when the remote operation is taken over, the contents of the remote operation (the location of the work machine which is remotely operated, the contents of the work, and the like), and the like.
Then, for example, by searching the database, the operator who is not in the other work machine is remotely operated, and the more the number of times the operator has experienced the remote operation of the work machine in a state having a high degree of approximation to the designated state, the more preferentially the remote operation device operated by the operator is selected as the 2 nd remote operation device 20. Additionally or alternatively, there may be an operator who is not remotely operating another work machine, and the more the number of times the operator takes over the remote operation of the work machine in this state, the more preferentially the remote operation device operated by the operator is selected as the 2 nd remote operation device 20. When the operator 2 is selected, the indexes are digitized as scores, and one or more remote operation devices of the operator whose score is equal to or larger than a reference value are selected as the operator 2 remote operation device 20.
In addition to the above, it is also possible to register in advance the skill levels of a plurality of 2 nd operators corresponding to the work contents (excavation, loading, leveling on level ground, forming on level ground, leveling on a sloping surface, and forming on a sloping surface) and preferentially select an operator having a higher skill level of an operation required for the operation of the takeover. The 2 nd notification corresponding to the 1 st notification is transmitted to the selected one or more 2 nd remote operation devices 20 by the 2 nd server arithmetic processing element 32 through the wireless communication machine (fig. 5/step 306). The "2 nd notification" may include, as in the case of the 1 st notification, an environment image indicating the environment of the work machine 40 immediately before the detection of the specified state, information necessary for the remote operation of the work machine 40 such as voice information, a construction plan, and an operation state and type of the work machine 40, and may further include a message requesting the remote operation to be taken over.
In the 2 nd remote control device 20, the 2 nd notification is received by the 2 nd client control device 200 through the 2 nd wireless communication device 222 (fig. 5/step 202). The 2 nd notification information corresponding to the 2 nd notification is displayed on the 2 nd image output apparatus 221 by the 2 nd client control apparatus 200 (fig. 5/step 204). For example, an environment image of work machine 40 (see fig. 6) and an image showing an operation state such as an inclination angle of upper revolving unit 420 of work machine 40 with respect to a horizontal plane are displayed on 2 nd image output device 221 as 2 nd notification information.
With the 2 nd client control device 200, it is determined whether the 2 nd operator agrees to take over the remote operation (or agrees and refuses to distinguish) by the operation of the 2 nd input interface 210 (fig. 5/step 206). If the determination result is negative (no in fig. 5/step 206), the process in the 2 nd client control apparatus 200 is ended. On the other hand, if the determination result is affirmative (y in fig. 5/step 206), the No. 3 notification is transmitted to the remote operation server 30 through the No. 2 wireless communication device 222 by the No. 2 client control device 200 (fig. 5/step 208). The "3 rd notification" includes, in addition to the fact that the 2 nd operator has agreed to take over the remote operation of the work machine 40, an identifier for identifying the 2 nd remote operation device 20.
The No. 3 notification is received by the No. 2 server computing processing element 32 through the wireless communication machine (FIG. 5/step 306). The 4 th notification corresponding to the 3 rd notification is transmitted to the 1 st remote operation device 10 by the 2 nd server arithmetic processing element 32 through the wireless communication device (fig. 5/step 308). The "4 th notification" includes the 2 nd operator who has agreed to take over the remote operation or the 2 nd remote operation device 20, and also includes the location of the work device remotely operated by the 2 nd operator, the work content, and the like. If the 3 rd notification indicates that the 2 nd operator does not agree with the takeover, the remote operation authority of the work machine 40 may be changed to the 2 nd remote operation device 20 without using the 2 nd server arithmetic processing element 32.
In the 1 st remote operation device 10, the 4 th notification is received by the 1 st client control device 100 via the 1 st wireless communication device 122 (fig. 5/step 118). The 1 st client control device 100 displays the 4 th notification information corresponding to the 4 th notification on the 1 st image output device 121 (fig. 5/step 120). For example, a list of the 2 nd operators (operation subject candidates) who have agreed to remotely operate the work machine 40 is displayed on the 2 nd image output device 221 as the 4 th notification information together with attributes such as the contents of their remote operation experiences.
With the 1 st client control device 100, the 2 nd operator designated by the 1 st operator through the 1 st input interface 110 as an operation subject candidate is identified (fig. 5/step 122). The 5 th notification corresponding to the selection result is transmitted to the remote operation server 30 via the 1 st wireless communication device 122 by the 1 st client control device 100 (fig. 5/step 124). The "5 th notice" contains an identifier for identifying the 2 nd operator designated by the 1 st operator.
In the remote operation server 30, the 5 th notification is received by the 2 nd server arithmetic processing element 32 through the wireless communication machine (fig. 5/step 312).
The 6 th notification is transmitted to the 2 nd remote operation device 20 of the 2 nd operator designated by the 1 st operator by the 2 nd server arithmetic processing element 32 through the wireless communication machine (fig. 5/step 316). The "notification 6" includes a determination result of whether or not to take over the operation. In the 2 nd remote operation device 20, the 6 th notification is received by the 2 nd client control device 200 through the 2 nd wireless communication device 222 (fig. 5/step 210). If the determination result included in "notification 6" is negative (no in fig. 5/step 212), the process in the 2 nd client control apparatus 200 is ended. If the determination result included in the "notification 6 th" is affirmative, the 2 nd remote operation device 20 is allowed to transmit the operation command to the work machine 40. The 2 nd remote operation device 20 becomes a new 1 st remote operation device 10 capable of remotely operating the working machine 40 (see fig. 4). That is, when the 6 th notification is a notification having a content for changing the remote operation authority, the 2 nd client control apparatus 200 acquires the remote operation authority of the work machine 40 from the 1 st remote operation apparatus 10.
Accordingly, with the 2 nd server arithmetic processing element 32, processing for changing the remote operation authority of the work machine 40 from the 1 st remote operation device 10 to the 2 nd remote operation device 20 is executed (fig. 5/step 316). For example, transmission of an operation command to the work machine 40 by the 1 st remote operation device 10 is prohibited (fig. 5/step 126). The 1 st remote operation device 10 is again brought into a state in which it can take over the remote operation of the working machine with respect to the 2 nd remote operation device 20. Then, the 2 nd remote operation device 20 acquires the operation authority of the work machine 40 (fig. 5/step 214).
When the takeover operation by the 2 nd remote operation device 20 of the work machine 40 is completed (fig. 5/step 216), a process of restoring the remote operation authority of the work machine 40 is executed (fig. 5/step 318). For example, the 1 st remote operation device 10 resumes sending the operation command to the work machine 40 (fig. 5/step 128). Further, transmission of an operation command to the work machine 40 by the 2 nd remote operation device 20 is prohibited (fig. 5/step 218).
(Effect)
According to the remote operation system configured as described above, when the work machine 40 is remotely operated by the 1 st remote operation device 10, if one or both of the states of the work machine 40 and the 1 st operator are in the designated state, the 1 st notification corresponding to the designated state is referred to, and the appropriate 2 nd remote operation device 20 can be selected as the remote operation subject for the work machine 40 (see fig. 5). This makes it possible to avoid instability in the operating state of the work machine 40 when switching the remote operation body of the work machine 40.
Further, since the notification information corresponding to the 2 nd notification is displayed on the image output device 221 constituting the 2 nd remote operation device 20, information necessary for switching the remote operation body of the work machine 40 can be provided to the operator of the 2 nd remote operation device 20 (see fig. 5/step 204).
(Another embodiment of the present invention)
In the above embodiment, the remote operation subject of the work machine 40 is switched as a requirement that the 1 st operator and the 2 nd operator agree to take over of the remote operation of the work machine 40 (see fig. 5/step 206, step 122, and step 314), but as another embodiment, the remote operation subject of the work machine 40 may be switched even if there is no agreement by one or both of the 1 st operator and the 2 nd operator.
It is also possible to provide a terminal (personal computer, smartphone, tablet, etc.) which is attached to the 2 nd remote operation device 20 and has the functions of the 2 nd input interface 210 and the 2 nd output interface 220 in cooperation with the 2 nd remote operation device 20 outside the 2 nd remote operation device 20, and to make the terminal have the functions of transmitting and receiving information such as the 2 nd notification, the 3 rd notification, and the 6 th notification, and the functions of the 2 nd input interface 210 and the 2 nd output interface.
In this case, for example, even in a place distant from the place where the 2 nd remote operation device 20 is installed, the 2 nd operator can receive information from the remote operation server 30 and perform the operation takeover operation. Specifically, the 2 nd notification and the 6 th notification transmitted from the remote operation server 30 can be received by mail, SNS, or the like, and the 2 nd operator can confirm information necessary when the work machine 40 is remotely operated from the notification information, determine whether or not to approve the operation takeover operation, and notify the remote operation server 30 of the approval. Further, the process up to the 6 th notification (information on whether or not to perform the operation takeover) can be performed at a place remote from the 2 nd remote operation device 20. The user can go to the 2 nd remote operation device 20 after confirming the takeover determined to perform the operation.
Although the "2 nd notification" in the above embodiment is configured to include information necessary for remote operation of the work machine 40 immediately before the specified state is detected, the "2 nd notification" may include only information indicating that the specified state is detected or information indicating that the specified state is the minimum, and may transmit information necessary for remote operation of the work machine 40 immediately before the specified state is detected to the 2 nd remote operation device 20 separately.
In this case, for example, only the 1 st remote operation device 10 in which the designated state is detected is notified to the 2 nd operator who operates the 2 nd remote operation device 20. Next, when the 2 nd operator considers the operation takeover, the information necessary for the remote operation of the work machine 40 immediately before the detection of the designated state is acquired by the remote operation server 30. With such a configuration, it is possible to provide more detailed information to the 2 nd operator who can perform the takeover operation with a remaining force.
In the above embodiment, when the number of candidates for the 2 nd operator is small, the score may be limited to a candidate having a reference value or more, and the 1 st operator and the 2 nd operator may select the 2 nd operator who performs the takeover operation by direct communication such as mail, chat on the internet, or telephone. In this case, the 4 th notification, which is the list information of the 2 nd operators (operation subject candidates) who have agreed to remotely operate the work machine 40, may be transmitted to the 2 nd operators who have agreed to remotely operate the work machine 40.
The 1 st server arithmetic processing element 31 may select the candidate of the 2 nd remote operation device 20 as the candidate of the target of the remote operation authority of the work machine 40 which is changed from the 1 st remote operation device 10, based on the 3 rd notification transmitted by the plurality of 2 nd remote operation devices 20. According to the remote operation system having this configuration, one 2 nd remote operation device 20 to which the remote operation authority of the work machine 40 is to be changed can be selected based on the plurality of 3 rd notifications transmitted from the plurality of 2 nd remote operation devices 20, respectively.
Description of the reference numerals
10. 1 st remote operation device
20. 2 nd remote operation device
21. 2 nd input interface
22. 2 nd output interface
25. Terminal device
30. Remote operation server
31. 1 st server arithmetic processing element
32. 2 nd server arithmetic processing element
40. Working machine
100. 1 st client control device
110. No. 1 input interface
111. 1 st operating mechanism
112. State detector
120. No. 1 output interface
121. 1 st image output device
122. 1 st radio communication device
200. 2 nd client control device
210. No. 2 input interface
211. No. 2 operating mechanism
220. 2 nd output interface
221. 2 nd image output device
222. 2 nd radio communication device
401. Image capturing apparatus
402. Wireless communication machine
440. Work attachment (work mechanism).
Claims (7)
1. A remote operation system is provided with: a working machine having a working mechanism and an imaging device that images an environment including at least a part of the working mechanism; a plurality of remote operation devices each having an operation mechanism for remotely operating the work machine and an image output device for displaying an environment image representing the environment of the work machine captured by the imaging device; a remote operation server having a function of communicating with each of the plurality of remote operation devices,
the 1 st remote operation device of the plurality of remote operation devices includes: a state detector that detects a state of at least one of the work machine and the operator as a specified state; a 1 st client control device that, when the state detector detects that the state of at least one of the work machine and the operator is the designated state, transmits a 1 st notification corresponding to the designated state,
the remote operation server includes: a 1 st server arithmetic processing element that receives the 1 st notification and selects a 2 nd remote operation device different from the 1 st remote operation device among the plurality of remote operation devices based on the 1 st notification; a 2 nd server arithmetic processing element that transmits a 2 nd notification corresponding to the 1 st notification to the 2 nd remote operation device or a 2 nd client control device included in the 2 nd remote operation device, and executes processing for changing a remote operation authority of the work machine from the 1 st remote operation device to the 2 nd remote operation device,
the 2 nd client control apparatus receives the 2 nd notification, and causes the image output apparatus to output notification information corresponding to the 2 nd notification.
2. The remote operating system of claim 1,
the image output device is a constituent element of a terminal device having a function of cooperating with the 2 nd remote operation device.
3. The remote operating system of claim 1,
the 2 nd client control device transmitting, as a 3 rd notification, a notification to the remote operation server indicating whether or not it is approved to change the remote operation authority of the work machine from the 1 st remote operation device to the 2 nd remote operation device, based on the received 2 nd notification,
the 2 nd server arithmetic processing element does not change the remote operation authority of the work machine to the 2 nd remote operation device when the 3 rd notification is a notification indicating that the agreement is not present.
4. The remote operating system of claim 3,
the 3 rd notification may be transmitted by a terminal device having a function cooperating with the 2 nd remote operation device.
5. The remote operation system according to claim 3 or 4,
the 1 st server arithmetic processing element selects the candidate of the 2 nd remote operation device as the candidate of the target of the remote operation authority of the working machine to be changed from the 1 st remote operation device, based on the 3 rd notification transmitted from the plurality of 2 nd remote operation devices.
6. The remote operation system according to claim 5,
said 1 st server arithmetic processing element selecting one of said 2 nd remote operation devices from among said candidates,
the 2 nd server arithmetic processing element notifies a plurality of the 2 nd remote operation devices of a notification indicating whether or not the remote operation authority is changed as a 6 th notification from the remote operation server,
the 2 nd client control device acquires the remote operation authority of the work machine from the 1 st remote operation device when the 6 th notification is a notification having a content of change of the remote operation authority.
7. A remote operation server having a function of communicating with a plurality of remote operation devices for remotely operating a working machine, respectively, the remote operation server comprising:
a 1 st server arithmetic processing element that receives, by a 1 st remote operation device among the plurality of remote operation devices, information corresponding to a specified state and a 1 st notification that is an environment image showing an environment of the work machine immediately before the specified state is detected, the information being transmitted from the 1 st remote operation device, and that selects a 2 nd remote operation device, which is different from the 1 st remote operation device, among the plurality of remote operation devices, in accordance with the 1 st remote operation device that detects that the state of at least one of the work machine and the operator is the specified state;
and a 2 nd server arithmetic processing element that transmits a 2 nd notification corresponding to the 1 st notification to the 2 nd remote operation device and executes processing for changing the remote operation authority of the work machine from the 1 st remote operation device to the 2 nd remote operation device.
Applications Claiming Priority (3)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| JP2019-059760 | 2019-03-27 | ||
| JP2019059760A JP7287047B2 (en) | 2019-03-27 | 2019-03-27 | Remote control system and remote control server |
| PCT/JP2019/047071 WO2020194884A1 (en) | 2019-03-27 | 2019-12-02 | Remote operation system and remote operation server |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113614317A CN113614317A (en) | 2021-11-05 |
| CN113614317B true CN113614317B (en) | 2022-12-09 |
Family
ID=72609372
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201980094527.6A Active CN113614317B (en) | 2019-03-27 | 2019-12-02 | Remote operation system and remote operation server |
Country Status (5)
| Country | Link |
|---|---|
| US (1) | US11885101B2 (en) |
| EP (1) | EP3922775B1 (en) |
| JP (1) | JP7287047B2 (en) |
| CN (1) | CN113614317B (en) |
| WO (1) | WO2020194884A1 (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR102538666B1 (en) * | 2020-12-04 | 2023-06-01 | 주식회사 팀그릿 | Remote control method for radio control toy and apparatus therefor |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP0976879A1 (en) * | 1997-10-29 | 2000-02-02 | Shin Caterpillar Mitsubishi Ltd. | Remote radio operating system, and remote operating apparatus, mobile relay station and radio mobile working machine |
| JP2006301723A (en) * | 2005-04-15 | 2006-11-02 | Toyota Motor Corp | Remote operation system |
| CN104838394A (en) * | 2013-12-11 | 2015-08-12 | 株式会社小松制作所 | Working machine management method, management computer program, and management system |
| CN107111928A (en) * | 2014-11-17 | 2017-08-29 | 洋马株式会社 | Remotely controlled operation machine display system |
Family Cites Families (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9715008B1 (en) * | 2013-03-20 | 2017-07-25 | Bentley Systems, Incorporated | Visualization of 3-D GPR data in augmented reality |
| US10503249B2 (en) * | 2014-07-03 | 2019-12-10 | Topcon Positioning Systems, Inc. | Method and apparatus for construction machine visualization |
| KR102250821B1 (en) * | 2014-08-20 | 2021-05-11 | 삼성전자주식회사 | Display apparatus and operating method thereof |
| JP2016076801A (en) | 2014-10-03 | 2016-05-12 | ヤンマー株式会社 | Remote monitoring system |
| WO2017043108A1 (en) * | 2015-09-10 | 2017-03-16 | 富士フイルム株式会社 | Projection-type display device and projection control method |
| US10242562B2 (en) * | 2017-02-22 | 2019-03-26 | Thames Technology Holdings, Inc. | Control systems with modular configurable devices |
| JP6807781B2 (en) * | 2017-03-13 | 2021-01-06 | 株式会社小松製作所 | Display system, display method, and remote control system |
| US20210140147A1 (en) * | 2017-06-13 | 2021-05-13 | Volvo Construction Equipment Ab | A working machine provided with an image projection arrangement |
| KR102659077B1 (en) * | 2018-03-27 | 2024-04-18 | 스미도모쥬기가이고교 가부시키가이샤 | shovel |
| KR20210021945A (en) * | 2018-06-19 | 2021-03-02 | 스미토모 겐키 가부시키가이샤 | Excavator, information processing device |
| CN112996963B (en) * | 2018-10-31 | 2022-12-02 | 住友建机株式会社 | Shovel and shovel support system |
| EP3879036A4 (en) * | 2018-11-08 | 2022-01-05 | Sumitomo Construction Machinery Co., Ltd. | Shovel, information processing device, information processing method, information processing program, terminal device, display method, and display program |
| EP3934241A4 (en) * | 2019-02-28 | 2022-04-20 | Sumitomo Heavy Industries, Ltd. | Display device, shovel, information processing device |
-
2019
- 2019-03-27 JP JP2019059760A patent/JP7287047B2/en active Active
- 2019-12-02 CN CN201980094527.6A patent/CN113614317B/en active Active
- 2019-12-02 EP EP19921240.8A patent/EP3922775B1/en active Active
- 2019-12-02 US US17/438,642 patent/US11885101B2/en active Active
- 2019-12-02 WO PCT/JP2019/047071 patent/WO2020194884A1/en unknown
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP0976879A1 (en) * | 1997-10-29 | 2000-02-02 | Shin Caterpillar Mitsubishi Ltd. | Remote radio operating system, and remote operating apparatus, mobile relay station and radio mobile working machine |
| JP2006301723A (en) * | 2005-04-15 | 2006-11-02 | Toyota Motor Corp | Remote operation system |
| CN104838394A (en) * | 2013-12-11 | 2015-08-12 | 株式会社小松制作所 | Working machine management method, management computer program, and management system |
| CN107111928A (en) * | 2014-11-17 | 2017-08-29 | 洋马株式会社 | Remotely controlled operation machine display system |
| JP2018501684A (en) * | 2014-11-17 | 2018-01-18 | ヤンマー株式会社 | Display system for remote control of work equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| US20220186466A1 (en) | 2022-06-16 |
| EP3922775A1 (en) | 2021-12-15 |
| JP7287047B2 (en) | 2023-06-06 |
| EP3922775A4 (en) | 2022-04-13 |
| CN113614317A (en) | 2021-11-05 |
| WO2020194884A1 (en) | 2020-10-01 |
| US11885101B2 (en) | 2024-01-30 |
| EP3922775B1 (en) | 2023-07-12 |
| JP2020161996A (en) | 2020-10-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2020095945A1 (en) | Shovel, information processing device, information processing method, information processing program, terminal device, display method, and display program | |
| CN115003888B (en) | Remote operation-assisted server, method and system | |
| JP7472578B2 (en) | REMOTE OPERATION SUPPORT SERVER, REMOTE OPERATION SUPPORT SYSTEM, AND REMOTE OPERATION SUPPORT METHOD | |
| US11993922B2 (en) | Remote operation system | |
| CN113614317B (en) | Remote operation system and remote operation server | |
| CN113615164B (en) | Remote operation system and remote operation server | |
| CN115552501A (en) | Remote operation support device and remote operation support method | |
| US20220389683A1 (en) | Work assist server, work assist method, and work assist system | |
| CN115298717A (en) | Obstacle reporting system for work machine and obstacle reporting method for work machine | |
| CN115362295B (en) | Remote operation support server, remote operation support system, and remote operation support method | |
| WO2021106281A1 (en) | Work assist server, work assist method, and work assist system | |
| US20220398512A1 (en) | Work assist server, work assist method, and work assist system | |
| CN115698438A (en) | Real machine state monitoring system and real machine state monitoring method | |
| JP2021130932A (en) | Remote operation device, remote operation assistance server, remote operation assistance system, and remote operation assistance method | |
| CN114902311A (en) | Work support server, work support method, and work support system | |
| JP2023150082A (en) | Support device, work machine and program |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |