WO2023189631A1 - Work machine and remote-controlled work machine system - Google Patents

Work machine and remote-controlled work machine system Download PDF

Info

Publication number
WO2023189631A1
WO2023189631A1 PCT/JP2023/010201 JP2023010201W WO2023189631A1 WO 2023189631 A1 WO2023189631 A1 WO 2023189631A1 JP 2023010201 W JP2023010201 W JP 2023010201W WO 2023189631 A1 WO2023189631 A1 WO 2023189631A1
Authority
WO
WIPO (PCT)
Prior art keywords
video
data
control device
working
video area
Prior art date
Application number
PCT/JP2023/010201
Other languages
French (fr)
Japanese (ja)
Inventor
哲平 齋藤
理優 成川
匡士 小谷
英明 伊東
慧 佐藤
英史 石本
Original Assignee
日立建機株式会社
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 日立建機株式会社 filed Critical 日立建機株式会社
Publication of WO2023189631A1 publication Critical patent/WO2023189631A1/en

Links

Images

Classifications

    • E—FIXED CONSTRUCTIONS
    • E02—HYDRAULIC ENGINEERING; FOUNDATIONS; SOIL SHIFTING
    • E02F—DREDGING; SOIL-SHIFTING
    • E02F9/00—Component parts of dredgers or soil-shifting machines, not restricted to one of the kinds covered by groups E02F3/00 - E02F7/00
    • E02F9/20—Drives; Control devices

Definitions

  • the control device for a working machine disclosed in Patent Document 1 calculates the communication delay time of video data based on the time when the video data was sent to the remote control device and the time when the video data returned from the remote control device was received. Then, the operation amount indicated by the operation signal received from the remote control device is multiplied by a gain corresponding to the communication delay time of the video data to limit the operation of the working machine.
  • the amount of video data i.e., the amount of communication
  • the amount of data is uniformly reduced for the entire video data, that is, if the resolution is uniformly reduced for the entire video data, it is possible to reduce not only the video area that is not necessary for the operation of the work machine, but also the The image area necessary for the operation of the machine also becomes difficult to see, reducing the productivity of the work machine.
  • FIG. 1 is a schematic diagram showing the configuration of a remote-controlled shovel system according to a first embodiment of the present invention.
  • FIG. 1 is a block diagram showing the functional configuration of a control device for an excavator in a first embodiment of the present invention together with related equipment.
  • FIG. 2 is a block diagram showing the functional configuration of a control device of a remote control device in a first embodiment of the present invention together with related equipment. It is a flow chart showing the procedure of video data compression processing of a control device of an excavator in a 1st embodiment of the present invention.
  • FIG. 3 is a diagram for explaining a specific example of video data compression processing of the excavator control device according to the first embodiment of the present invention.
  • FIG. 1 is a block diagram showing the functional configuration of a control device for an excavator in a first embodiment of the present invention together with related equipment.
  • FIG. 2 is a block diagram showing the functional configuration of a control device of a remote control device in a first embodiment of
  • FIG. 7 is a diagram for explaining a specific example of video data compression processing of the excavator control device in a modified example of the present invention.
  • FIG. 7 is a diagram for explaining a specific example of video data compression processing of the excavator control device in a modified example of the present invention. It is a block diagram showing the functional composition of the control device of the excavator in the 2nd embodiment of the present invention together with related equipment. It is a block diagram showing the functional composition of the control device of the excavator in the 3rd embodiment of the present invention together with related equipment.
  • the excavator 1 includes a traveling body 3 that can travel, a rotating body 4 that is rotatably provided above the traveling body 3, and an articulated working device that is provided on the front side of the rotating body 4 (on the left side in FIG. 1). 5.
  • the traveling body 3 travels by rotation of a traveling motor (not shown), and the swinging body 4 turns by rotation of a swing motor (not shown).
  • the working device 5 includes a boom 6 rotatably connected to the revolving structure 4, an arm 7 rotatably connected to the tip of the boom 6, and a boom 6 rotatably connected to the tip of the arm 7.
  • a bucket 8 (work tool) is provided.
  • the boom 6 rotates as the boom cylinder 9 expands and contracts
  • the arm 7 rotates as the arm cylinder 10 expands and contracts
  • the bucket 8 rotates as the bucket cylinder 11 expands and contracts.
  • the excavator 1 has a photographing device 13 that takes images of the working device 5 and its surroundings, and a rotating angle of the rotating body 4, a rotating angle of the boom 6, a rotating angle of the arm 7, and a rotating angle of the bucket 8. It includes a plurality of angle sensors 14 for detection (see FIG. 2), a communication device 15 for communicating with the remote control device 2, and a control device 16.
  • the control device 16 has a processor that executes processing according to a program, a memory that stores programs and data, and the like.
  • the control device 16 has a functional configuration including an operation signal receiving section 17, a drive control section 18, a communication quality receiving section 19, a motion recognition section 20, a compression setting section 21, a video data compression section 22, and a video data transmission section 23.
  • the operation signal receiving unit 17 of the control device 16 receives operation signals from the remote control device 2 via the communication device 15 (in detail, the running of the traveling body 3, the rotation of the rotating body 4, the rotation of the boom 6, and the rotation of the arm 7). or an operation signal instructing the rotation of the bucket 8).
  • the drive control section 18 of the control device 16 controls the drive device 12 (specifically, the corresponding electromagnetic proportional valve) according to the operation signal received by the operation signal reception section 17. This causes the traveling body 3 to travel, the revolving body 4 to rotate, the boom 6 to rotate, the arm 7 to rotate, or the bucket 8 to rotate.
  • the communication quality receiving unit 19 of the control device 16 receives the communication quality evaluation value (evaluation result) from the remote control device 2 via the communication device 15.
  • the motion recognition unit 20 of the control device 16 recognizes the motion of the working device 5 (specifically, the positions of the boom 6, arm 7, and bucket 8, and the moving direction of the bucket 8) based on the detection results of the plurality of angle sensors 14.
  • the operation of the rotating body 4 (specifically, the turning direction of the rotating body 4) is recognized.
  • the compression setting section 21 of the control device 16 sets the compression setting section 21 in the photographing device 13 based on the communication quality evaluation value received by the communication quality receiving section 19 and the motion of the working device 5 or the rotating body 4 recognized by the motion recognition section 20.
  • the compression ratio of data indicating the photographed video (hereinafter referred to as video data) is set (details will be described later).
  • the video data compression unit 22 of the control device 16 generates video data for transmission by compressing the video data based on the compression rate set by the compression setting unit 21.
  • the video data transmitter 23 of the control device 16 transmits the video data for transmission generated by the video data compressor 22 to the remote control device 2 via the communication device 15.
  • the remote control device 2 includes a communication device 27 for communicating with the shovel 1 and a control device 28.
  • the control device 28 includes a processor that executes processing according to a program, a memory that stores programs and data, and the like.
  • the control device 28 has a video data reception section 29, a display control section 30, a communication quality evaluation section 31, a communication quality transmission section 32, and an operation signal transmission section 33 as functional components.
  • the communication quality evaluation section 31 of the control device 28 in the remote control device 2 determines the communication quality between the excavator 1 and the remote control device 2 based on the video data received by the video data receiving section 29. Evaluate quality. To explain in detail, the communication quality evaluation unit 31 calculates the delay time required for communication of video data, for example, based on the difference between the transmission cycle (fixed value) and the reception cycle (variable value) of the video data. Then, an evaluation value of communication quality (specifically, the lower the communication quality, the smaller the value) is calculated based on the communication delay time and packet loss of the video data. The communication quality transmitting unit 32 of the control device 28 transmits the communication quality evaluation value calculated by the communication quality evaluation unit 31 to the excavator 1 via the communication device 27.
  • FIG. 4 is a flowchart showing the procedure of video data compression processing of the excavator control device in this embodiment.
  • step S1 the communication quality receiving unit 19 of the control device 16 receives the communication quality evaluation value from the remote control device 2 via the communication device 15.
  • step S2 the compression setting unit 21 of the control device 16 determines whether the communication quality evaluation value received by the communication quality receiving unit 19 is less than a preset threshold.
  • step S3 the compression setting unit 21 of the control device 16 sets a standard compression rate so that the total amount of video data becomes a standard value.
  • the video data compression unit 22 of the control device 16 compresses video data representing videos shot by the imaging device 13 (in this embodiment, videos A1 to A5 shot by five cameras) at a standard compression rate. The entire video data is uniformly compressed to generate video data for transmission. Then, the video data transmitter 23 transmits the compressed video data for transmission.
  • step S6 the compression setting unit 21 of the control device 16 divides (divides) the video into a plurality of video regions, and also creates a plurality of video data (hereinafter referred to as video region data) corresponding to the plurality of video regions. Set a data reduction priority for each. Thereafter, in step S7, the compression setting unit 21 selects a plurality of video data based on the data reduction priority set in step S6 so as to achieve the total data reduction amount of the video data set in step S4. Set the data reduction amount or reduction rate of the area data.
  • the compression setting unit 21 determines the amount of data reduction (compression rate) of the video data to be displayed on the display device 24 based on the operating status of the work device 5 and the communication status with the remote control device 2 in units of video areas of the video. Set individually.
  • the compression setting unit 21 of the control device 16 recognizes the current or previous operation of the working device 5 and selects the image data of the working device 5 from among the video data corresponding to the images A1 to A5 shot by the five cameras. Select the video data in which bucket 8 appears. For example, as shown in FIG. 5, the video data corresponding to the selected video A1 is combined with the main video area data corresponding to the main video area R1 showing the bucket 8 of the work device 5 and its surroundings, and the boom of the work device 5. The video area data is divided into video area data corresponding to the video area R2 indicating 6 and sub-video area data corresponding to another video area R3.
  • the video data corresponding to the remaining videos A2 to A5 is the same as, for example, the video data corresponding to video A1. That is, for example, if the boom 6 of the working device 5 does not appear, the processing is the same as the video area data corresponding to the video area R3 of the video A1.
  • the first data reduction priority is video area data corresponding to video area R2 of video A1
  • the second is video area data corresponding to video area R3 of video A1 and video area data corresponding to videos A2 to A5.
  • the third is set as video area data corresponding to video area R1 of video A1.
  • step S4 based on the data reduction priority order described above, corresponding to video regions R1, R2, R3 of video A1 and videos A2 to A5, so as to achieve the data reduction amount of the entire video data set in step S4. Set the amount of data reduction for each video area data.
  • step S7 the video data compression unit 22 of the control device 16 reduces the amount of data of the video region data, which is the video data for each video region set in step S6, to reduce the amount of video captured by the imaging device 13. Generates video data for transmission that is non-uniformly compressed.
  • the video data transmitter 23 transmits the compressed video data for transmission.
  • a plurality of video regions including a main video region R1 showing the bucket 8 of the work device 5 and its surroundings and other video regions are created according to the operation of the work device 5 or the operation of the revolving structure 4.
  • main video area data that is video area data corresponding to the main video area R1 and sub-video area data corresponding to other video areas are generated.
  • the data reduction priority order is set so that the sub-video area data is prioritized over the main video area data. For example, data reduction is performed such that the reduction amount or reduction rate of the sub-video area data is greater than that of the main video area data, or the data amount of the sub-video area data is larger than the data amount of the main video area data.
  • the control device 16 of the shovel 1 moves the center of the part of the bucket 8 to be watched (specifically, the opening and claw of the bucket 8) to the center of the ellipse.
  • the main video region R1 is set as an elliptical region
  • the present invention is not limited to this.
  • the control device 16 may set the main image region R1 to be an elliptical region with the center of the portion of the bucket 8 to be watched as the focal point of the ellipse. That is, as shown in FIG.
  • the main video area R1 when the bucket 8 moves downward, the main video area R1 is adjusted so that the area on the side of the moving direction is larger than the area on the opposite side with respect to the area of the bucket 8.
  • the shape and range may be set.
  • the main video area R1 when the bucket 8 moves to the left, the main video area R1 may be set in such a shape that the area on the left is larger than the area on the right based on the area of the bucket 8. good.
  • the control device 16 divides the image into a plurality of image areas including a main image area indicating the movement destination of the working device 5 and other image areas, and Compression processing may be performed on video area data (main video area data and sub-video area data, respectively) corresponding to the divided video areas. That is, for example, when the working device 5 moves to the left as the rotating body 4 turns to the left, video data corresponding to videos A1 to A5 taken by five cameras is transferred to video A4 (main video area).
  • the main video area data corresponding to video A4 is divided into main video area data corresponding to other video A1-A3, A5 (other video areas), and the main video area data corresponding to video A4 is divided into video A1-A3, A5.
  • Data reduction priorities may be set so that corresponding sub-video area data is preferentially compressed. For example, when the working device 5 moves to the right as the revolving structure 4 turns to the right, video data corresponding to videos A1 to A5 taken by five cameras is transferred to video A5 (main video area).
  • the main video area data corresponding to the video A1-A4 (other video areas) are divided into the corresponding main video area data and the sub video area data corresponding to the other videos A1-A4 (other video areas), and the sub video area data corresponding to the videos A1-A4 is divided from the main video area data corresponding to the video A5.
  • the priority order of data reduction may be set to give priority to video area data.
  • FIG. 7 is a block diagram showing the functional configuration of the excavator control device in this embodiment together with related equipment.
  • parts equivalent to those in the first embodiment are given the same reference numerals, and description thereof will be omitted as appropriate.
  • the control device 16A of the excavator 1 of this embodiment includes an operation signal receiving section 17, a drive control section 18, a communication quality receiving section 19, a motion recognition section 20, and a compression setting section 21. , a video data compression section 22, and a video data transmission section 23.
  • the compression setting unit 21 of the control device 16A sets the overall data reduction amount of the video data when the communication quality evaluation value is less than the threshold value.
  • the video data in which the dump truck appears is selected from among the video data corresponding to the videos A1 to A5 taken by the five cameras.
  • the video data corresponding to the selected video A4 is divided into main video area data corresponding to the main video area showing the dump truck and its surroundings, and sub-video area data corresponding to other video areas. .
  • the video data corresponding to the remaining videos A1 to A3 and A5 is the same as, for example, the video data corresponding to video A4. That is, for example, if a dump truck does not appear, the data is handled in the same way as video area data corresponding to other video areas of video A4.
  • the first data reduction priority is the video area data corresponding to other video areas of video A4 and the video area data corresponding to videos A1 to A3, and A5, and the second is the video area data corresponding to the main video area of video A4.
  • the main video area of video A4, other video areas, and videos A1 to A3 and A5 are Set the amount of data reduction for the corresponding video area data.
  • FIG. 8 is a block diagram showing the functional configuration of the excavator control device in this embodiment together with related equipment.
  • parts equivalent to those in the first and second embodiments are denoted by the same reference numerals, and description thereof will be omitted as appropriate.
  • the control device 16B of the excavator 1 of this embodiment like the control device 16A of the second embodiment, includes an operation signal receiving section 17, a drive control section 18, a communication quality receiving section 19, a compression setting section 21, and a video data compression section. 22, a video data transmitter 23, and a moving object detector 34.
  • the control device 16B has a motion prediction section 36 in place of the motion recognition section 20 of the control device 16A of the second embodiment.
  • the motion prediction section 36 of the control device 16B like the motion recognition section 20, recognizes the motion of the working device 5 and the motion of the rotating structure 4 from the detection results of the plurality of angle sensors. Then, the operating pattern of the excavator 1 related to each work of the excavator 1 is learned and stored based on the transition of the operation of the working device 5 or the operation of the revolving structure 4, and whether or not a dump truck is detected by the moving object detection unit 34. Then, using the above-mentioned operation pattern, the future operation of the excavator is predicted from the current operation of the excavator or the current detection of the vehicle.
  • an explanation will be given using an operation pattern of the shovel related to the loading work of the shovel 1.
  • this operation pattern first, a dump truck appears at a predetermined location away from, for example, the left or right side of the working device 5 of the shovel 1. Thereafter, the revolving body 4 of the excavator 1 turns toward the dump truck. After that, the working device 5 of the shovel 1 operates.
  • the operation prediction unit 36 of the control device 16B predicts that the revolving body 4 of the excavator 1 will turn toward the dump truck when the moving object detection unit 34 detects a dump truck.
  • the compression setting unit 21 of the control device 16B selects the video data in which the dump truck appears from among the video data corresponding to the videos A1 to A5 shot by the five cameras. For example, the video data corresponding to the selected video A4 is divided into main video area data corresponding to the main video area showing the dump truck and its surroundings, and sub-video area data corresponding to other video areas. . Furthermore, the video data in which the bucket 8 of the working device 5 appears is selected from among the video data corresponding to the videos A1 to A5 taken by the five cameras.
  • the video data corresponding to the selected video A1 is combined with the main video area data corresponding to the main video area R1 showing the bucket 8 of the work equipment 5 and its surroundings, and the video area R2 showing the boom 6 of the work equipment 5. and sub-video area data corresponding to another video area R3 (see FIG. 5 or FIG. 6B described above).
  • the video data corresponding to the remaining videos A2, A3, and A5 is the same as, for example, the video data corresponding to video A1. That is, for example, if the boom 6 of the working device 5 does not appear, the processing is the same as the video area data corresponding to the video area R3 of the video A1.
  • the first data reduction priority is video area data corresponding to video area R2 of video A1
  • the second is video area data corresponding to video area R3 of video A1
  • the second is video area data corresponding to other video areas of video A4.
  • the third set is video area data corresponding to video area R1 of video A1 and video area data corresponding to the main video area of video A4. .
  • video areas R1, R2, R3 of video A1, video areas R2, and R3 of video A4 are The amount of data reduction for video area data corresponding to the main video area, other video areas, and videos A2, A3, and A5 is set, respectively.
  • the excavator 1 uses the rotation angle of the boom 6, the rotation angle of the arm 7, and the rotation angle of the bucket 8 as sensors for detecting state quantities related to the operation of the working device 5.
  • the excavator 1 may include a plurality of inertial measurement devices that respectively detect the inclination angle of the boom 6, the inclination angle of the arm 7, and the inclination angle of the bucket 8, for example. Then, the operation of the work device 5 may be recognized based on the detection results of the plurality of inertial measurement devices.
  • control device 16 of the excavator 1 recognizes the operation of the working device 5 and the operation of the rotating structure 4 based on the detection results of the plurality of angle sensors 14.
  • the control device 16 of the excavator 1 Based on the operation signal received from the remote control device 2, the operation of the working device 5 and the operation of the revolving structure 4 may be recognized.
  • the working machine is a large excavator
  • the present invention is not limited to this.
  • it may be a medium-sized or small-sized excavator, or it may be a working machine other than an excavator (specifically, a crane truck, etc.).
  • a first invention provides a working device, a photographing device for photographing an image of the working device and its surroundings, and transmitting the video photographed by the photographing device as video data via a first communication device; a working machine including a first control device that controls the operation of the working device in response to an operation signal received via a first communication device; a display device that displays various information; an operating device that generates the operating signal; and displaying an image corresponding to the video data received from the work machine via a second communication device on the display device, and the operating signal generated by the operating device.
  • a remote control device including a second control device that transmits the information to the work machine via the second communication device, wherein the second control of the remote control device
  • the device evaluates the quality of communication with the working machine and transmits the evaluation result to the working machine, and the first control device of the working machine controls the operation of the working machine based on the operation signal.
  • the video data is divided into a plurality of video region data corresponding to a plurality of video regions of the video, and a data reduction amount is set for each of the plurality of video region data, and the set Reduced video data for transmission is generated based on the amount of data reduction for each region data, and the video data for transmission is transmitted to the remote control device.
  • a second invention includes a working device, a photographing device for photographing an image of the working device and its surroundings, and transmitting the video photographed by the photographing device as video data via a first communication device; a working machine including a first control device that controls the operation of the working device in response to an operation signal received via a first communication device; a display device that displays various information; an operating device that generates the operating signal; and displaying an image corresponding to the video data received from the work machine via a second communication device on the display device, and the operating signal generated by the operating device.
  • a remote control device including a second control device that transmits the information to the work machine via the second communication device, wherein the second control of the remote control device
  • the device evaluates the quality of communication with the working machine and transmits the evaluation result to the working machine, and the first control device of the working machine controls the operation of the working machine based on the operation signal.
  • the video data is divided into a plurality of video region data corresponding to a plurality of video regions of the video, and a data reduction amount is set for each of the plurality of video region data, and the set Reduced video data for transmission is generated based on the amount of data reduction for each region data, and the video data for transmission is transmitted to the remote control device.
  • a video area that can be displayed on the remote control device that is directly connected to the work is Real-time, high-quality images can be displayed on a remote control device without sacrificing the image quality of the area, making it possible to suppress delays in operations and work and, ultimately, a decline in productivity of working machines.

Abstract

The present invention provides a work machine and a remote control system for the work machine, with which any decrease in productivity of the work machine can be minimized while any decrease in quality of communication between the working machine and a remote control device is minimized. An excavator comprises a work device, an image-capture device for capturing an image of the work device and the surroundings thereof, and a control device for transmitting the image captured by the image-capture device as image data and controlling the action of the work device in accordance with a received operation signal. On the basis of a result of evaluating communication quality and the action status of the work device, the control device divides the image data into a plurality of items of image region data corresponding to a plurality of image regions in the image and sets a data reduction amount for each of the plurality of items of image region data. Image data for transmission, which is reduced on the basis of a set data reduction amount for each of the plurality of items of image region data, is then generated, and the image data for transmission is transmitted.

Description

作業機械及び遠隔操作型作業機械システムWork machines and remote-controlled work machine systems
　本発明は、遠隔操作可能な作業機械、及び遠隔操作型作業機械システムに関する。 The present invention relates to a remotely controllable work machine and a remote control work machine system.
　特許文献１は、遠隔操作可能な作業機械と、作業機械から離れた場所に配置され、作業機械を遠隔操作する遠隔操作装置とを備えた遠隔操作型の作業機械システムを開示する。作業機械は、走行可能な走行体と、走行体の上側に旋回可能に設けられた旋回体と、旋回体の前側に連結された多関節型の作業装置と、旋回体の前方の映像を撮影する撮影装置と、制御装置とを備える。遠隔操作装置は、映像を表示する表示装置と、操作者の操作に応じて操作信号を生成する操作装置と、制御装置とを備える。 Patent Document 1 discloses a remote-controlled work machine system that includes a work machine that can be operated remotely and a remote control device that is placed at a location away from the work machine and remotely controls the work machine. The work machine includes a movable traveling body, a rotating body that is rotatably installed above the traveling body, an articulated work device connected to the front side of the rotating body, and a camera that captures images in front of the rotating body. and a control device. The remote control device includes a display device that displays images, a control device that generates a control signal in response to an operation by an operator, and a control device.
　作業機械の制御装置は、撮影装置で撮影された映像を映像データとして遠隔操作装置へ送信する。遠隔操作装置の制御装置は、作業機械から受信した映像データに対応する映像を表示装置に表示させる。遠隔操作装置の制御装置は、操作装置で生成された操作信号を作業機械へ送信する。作業機械の制御装置は、遠隔操作装置から受信した操作信号に応じて、走行体、旋回体、及び作業装置の動作を制御する。 The control device of the work machine transmits the video shot by the imaging device to the remote control device as video data. The control device of the remote control device causes the display device to display an image corresponding to the image data received from the working machine. The control device of the remote control device transmits the control signal generated by the control device to the work machine. A control device for a work machine controls operations of a traveling body, a revolving body, and a work device according to an operation signal received from a remote control device.
　作業機械と遠隔操作装置の間の通信回線としてインターネット等を利用すると、他の通信回線の影響を受けて、通信品質（詳細には、通信速度、遅延時間、パケットロスなどの通信性能によって決まるもの）が低下し、通信遅延が発生する可能性がある。映像データは、操作信号と比べ、データ量が多いから、通信遅延が発生しやすい。 When the Internet is used as a communication line between a work machine and a remote control device, it is affected by other communication lines and the communication quality (specifically, communication speed, delay time, packet loss, etc. determined by communication performance) ), which may cause communication delays. Since video data has a larger amount of data than operation signals, communication delays are more likely to occur.
　特許文献１の作業機械の制御装置は、遠隔操作装置へ映像データを送信した時刻と、遠隔操作装置から返送された映像データを受信した時刻とに基づき、映像データの通信遅延時間を算出する。そして、遠隔操作装置から受信した操作信号が示す動作量に、映像データの通信遅延時間に応じたゲインを乗算して、作業機械の動作を制限する。 The control device for a working machine disclosed in Patent Document 1 calculates the communication delay time of video data based on the time when the video data was sent to the remote control device and the time when the video data returned from the remote control device was received. Then, the operation amount indicated by the operation signal received from the remote control device is multiplied by a gain corresponding to the communication delay time of the video data to limit the operation of the working machine.
特開２０１９－０６８３４６号公報JP2019-068346A
　操作者は、遠隔操作装置の表示装置で表示された作業機械の周囲の映像を見ながら、操作装置を用いて作業機械を遠隔操作する。そのため、作業機械と遠隔操作装置の間の通信の品質が低下し、映像の表示遅延が生じた場合に、操作装置の操作も遅れることになり、作業機械の生産性が低下する。特許文献１では、映像の表示遅延が生じた場合に、映像の遅延を改善せず、作業機械の動作を制限するから、作業機械の生産性が更に低下する。 The operator remotely controls the work machine using the operating device while viewing an image of the surroundings of the work machine displayed on the display device of the remote control device. Therefore, if the quality of communication between the work machine and the remote control device is degraded and a delay in displaying images occurs, operation of the control device will also be delayed, reducing the productivity of the work machine. In Patent Document 1, when a video display delay occurs, the operation of the working machine is restricted without improving the video delay, which further reduces the productivity of the working machine.
　通信品質の低下が生じた場合に、通信品質の低下を抑えるため、映像データのデータ量（すなわち、通信量）を低減することが考えられる。しかし、映像データの全体に対して均一にデータ量を低減すれば、すなわち、映像データの全体に対して均一に解像度を低減すれば、作業機械の動作に必要でない映像領域だけでなく、作業機械の動作に必要な映像領域も見えにくくなり、作業機械の生産性が低下する。 In order to suppress the deterioration of communication quality when the deterioration of communication quality occurs, it is conceivable to reduce the amount of video data (i.e., the amount of communication). However, if the amount of data is uniformly reduced for the entire video data, that is, if the resolution is uniformly reduced for the entire video data, it is possible to reduce not only the video area that is not necessary for the operation of the work machine, but also the The image area necessary for the operation of the machine also becomes difficult to see, reducing the productivity of the work machine.
　本発明は、上記事柄に鑑みてなされたものであり、その目的は、作業機械と遠隔操作装置との間の通信品質が低下した場合でも遠隔操作装置において作業機械の動作状況及び作業状況を正確に把握することができ、作業機械の生産性の低下を抑えることができる作業機械及び遠隔操作型作業機械システムを提供することにある。 The present invention has been made in view of the above-mentioned problems, and an object of the present invention is to enable the remote control device to accurately determine the operational status and work status of the work machine even if the communication quality between the work machine and the remote control device is degraded. It is an object of the present invention to provide a working machine and a remote-controlled working machine system that can be grasped and suppress a decrease in the productivity of the working machine.
　上記目的を達成するために、本発明は、作業装置と、前記作業装置及びその周囲の映像を撮影する撮影装置と、前記撮影装置で撮影された映像を映像データとして通信装置を介して遠隔操作装置へ送信すると共に、前記遠隔操作装置から前記通信装置を介して受信した操作信号に応じて前記作業装置の動作を制御する制御装置と、を備えた作業機械において、前記制御装置は、前記遠隔操作装置から、前記通信装置を介して前記遠隔操作装置との間の通信品質の評価結果を受信し、前記操作信号に基づいて、前記作業装置の動作状況を判定し、前記通信品質の評価結果及び前記作業装置の動作状況に基づいて、前記映像データを前記映像の複数の映像領域に対応する複数の映像領域データに分割すると共に、前記複数の映像領域データのそれぞれに対するデータ低減量を設定し、当該設定された前記複数の映像領域データごとのデータ低減量に基づいて低減された送信用の映像データを生成し、当該送信用の映像データを前記通信装置を介して前記遠隔操作装置に送信する。 In order to achieve the above object, the present invention provides a working device, a photographing device that photographs images of the working device and its surroundings, and a remote control device that uses the photographed image of the photographing device as video data through a communication device. a control device that controls the operation of the work device according to an operation signal transmitted to the device and received from the remote control device via the communication device; Receive an evaluation result of communication quality with the remote control device from the operating device via the communication device, determine the operating status of the work device based on the operation signal, and determine the evaluation result of the communication quality. and dividing the video data into a plurality of video region data corresponding to a plurality of video regions of the video, and setting a data reduction amount for each of the plurality of video region data, based on the operating status of the work device. , generate reduced video data for transmission based on the set data reduction amount for each of the plurality of video area data, and transmit the video data for transmission to the remote control device via the communication device; do.
　本発明によれば、作業機械と遠隔操作装置との間の通信品質が低下した場合でも、例えば、遠隔操作装置に表示可能な映像領域のうちの作業に直結するような領域の映像品質を犠牲にすることなく、リアルタイムかつ高品質な映像を遠隔操作装置に表示させることができ、操作や作業の遅延、ひいては作業機械の生産性の低下を抑えることができる。 According to the present invention, even if the communication quality between the work machine and the remote control device is degraded, for example, the video quality of an area that is directly connected to the work out of the video area that can be displayed on the remote control device is sacrificed. It is possible to display real-time, high-quality images on a remote control device without having to use the system, and it is possible to suppress delays in operation and work, as well as decreases in the productivity of working machines.
本発明の第１の実施形態における遠隔操作型のショベルシステムの構成を表す概略図である。1 is a schematic diagram showing the configuration of a remote-controlled shovel system according to a first embodiment of the present invention. 本発明の第１の実施形態におけるショベルの制御装置の機能的構成を関連機器と共に表すブロック図である。FIG. 1 is a block diagram showing the functional configuration of a control device for an excavator in a first embodiment of the present invention together with related equipment. 本発明の第１の実施形態における遠隔操作装置の制御装置の機能的構成を関連機器と共に表すブロック図である。FIG. 2 is a block diagram showing the functional configuration of a control device of a remote control device in a first embodiment of the present invention together with related equipment. 本発明の第１の実施形態におけるショベルの制御装置の映像データ圧縮処理の手順を表すフローチャートである。It is a flow chart showing the procedure of video data compression processing of a control device of an excavator in a 1st embodiment of the present invention. 本発明の第１の実施形態におけるショベルの制御装置の映像データ圧縮処理の具体例を説明するための図である。FIG. 3 is a diagram for explaining a specific example of video data compression processing of the excavator control device according to the first embodiment of the present invention. 本発明の一変形例におけるショベルの制御装置の映像データ圧縮処理の具体例を説明するための図である。FIG. 7 is a diagram for explaining a specific example of video data compression processing of the excavator control device in a modified example of the present invention. 本発明の一変形例におけるショベルの制御装置の映像データ圧縮処理の具体例を説明するための図である。FIG. 7 is a diagram for explaining a specific example of video data compression processing of the excavator control device in a modified example of the present invention. 本発明の第２の実施形態におけるショベルの制御装置の機能的構成を関連機器と共に表すブロック図である。It is a block diagram showing the functional composition of the control device of the excavator in the 2nd embodiment of the present invention together with related equipment. 本発明の第３の実施形態におけるショベルの制御装置の機能的構成を関連機器と共に表すブロック図である。It is a block diagram showing the functional composition of the control device of the excavator in the 3rd embodiment of the present invention together with related equipment.
　本発明の第１の実施形態を、図面を参照しつつ説明する。本実施形態では、作業機械として、鉱山等で稼動する大型のショベルを例にとっている。 A first embodiment of the present invention will be described with reference to the drawings. In this embodiment, a large excavator operated in a mine or the like is taken as an example of a working machine.
　図１は、本実施形態における遠隔操作型のショベルシステムの構成を表す概略図である。図２は、本実施形態におけるショベルの制御装置の機能的構成を関連機器と共に表すブロック図である。図３は、本実施形態における遠隔操作装置の制御装置の機能的構成を関連機器と共に表すブロック図である。 FIG. 1 is a schematic diagram showing the configuration of a remote-controlled excavator system in this embodiment. FIG. 2 is a block diagram showing the functional configuration of the excavator control device in this embodiment together with related equipment. FIG. 3 is a block diagram showing the functional configuration of the control device of the remote control device in this embodiment together with related equipment.
　本実施形態のショベルシステム１００は、遠隔操作可能なショベル１と、ショベル１から離れた場所に配置され、ショベル１の外部に設けられてショベル１を遠隔操作によって操作するように構成された遠隔操作装置２とを備える。 The shovel system 100 of the present embodiment includes a shovel 1 that can be operated remotely, and a remote control that is arranged at a location away from the shovel 1 and is provided outside the shovel 1 and configured to operate the shovel 1 by remote control. A device 2 is provided.
　ショベル１は、走行可能な走行体３と、走行体３の上側に旋回可能に設けられた旋回体４と、旋回体４の前側（図１の左側）に設けられた多関節型の作業装置５とを備える。走行体３は、走行モータ（図示せず）の回転によって走行し、旋回体４は、旋回モータ（図示せず）の回転によって旋回する。 The excavator 1 includes a traveling body 3 that can travel, a rotating body 4 that is rotatably provided above the traveling body 3, and an articulated working device that is provided on the front side of the rotating body 4 (on the left side in FIG. 1). 5. The traveling body 3 travels by rotation of a traveling motor (not shown), and the swinging body 4 turns by rotation of a swing motor (not shown).
　作業装置５は、旋回体４に回動可能に連結されたブーム６と、ブーム６の先端部に回動可能に連結されたアーム７と、アーム７の先端部に回動可能に連結されたバケット８（作業具）とを備える。ブーム６は、ブームシリンダ９の伸縮によって回動し、アーム７は、アームシリンダ１０の伸縮によって回動し、バケット８は、バケットシリンダ１１の伸縮によって回動する。 The working device 5 includes a boom 6 rotatably connected to the revolving structure 4, an arm 7 rotatably connected to the tip of the boom 6, and a boom 6 rotatably connected to the tip of the arm 7. A bucket 8 (work tool) is provided. The boom 6 rotates as the boom cylinder 9 expands and contracts, the arm 7 rotates as the arm cylinder 10 expands and contracts, and the bucket 8 rotates as the bucket cylinder 11 expands and contracts.
　ショベル１は、複数の油圧アクチュエータ（詳細には、上述した走行モータ、旋回モータ、ブームシリンダ９、アームシリンダ１０、及びバケットシリンダ１１）を駆動する駆動装置１２（図２参照）を備える。駆動装置１２は、例えば、エンジンと、エンジンによって駆動される油圧ポンプ及びパイロットポンプと、油圧ポンプから複数の油圧アクチュエータへの圧油の流れを制御する複数の制御弁と、パイロットポンプの吐出圧を元圧として、複数の制御弁を操作するパイロット圧を生成する複数の電磁比例弁とを備える。 The excavator 1 includes a drive device 12 (see FIG. 2) that drives a plurality of hydraulic actuators (specifically, the above-mentioned travel motor, swing motor, boom cylinder 9, arm cylinder 10, and bucket cylinder 11). The drive device 12 includes, for example, an engine, a hydraulic pump and a pilot pump driven by the engine, a plurality of control valves that control the flow of pressure oil from the hydraulic pump to a plurality of hydraulic actuators, and a discharge pressure of the pilot pump. It is equipped with a plurality of electromagnetic proportional valves that generate pilot pressure, which operates a plurality of control valves, as a source pressure.
　ショベル１は、作業装置５及びその周囲の映像を撮影する撮影装置１３と、旋回体４の旋回角、ブーム６の回動角、アーム７の回動角、及びバケット８の回動角をそれぞれ検出する複数の角度センサ１４（図２参照）と、遠隔操作装置２との通信を行うための通信装置１５と、制御装置１６とを備える。 The excavator 1 has a photographing device 13 that takes images of the working device 5 and its surroundings, and a rotating angle of the rotating body 4, a rotating angle of the boom 6, a rotating angle of the arm 7, and a rotating angle of the bucket 8. It includes a plurality of angle sensors 14 for detection (see FIG. 2), a communication device 15 for communicating with the remote control device 2, and a control device 16.
　撮影装置１３は、本実施形態においては５つのカメラで構成されているが、その一部の図示を省略している。第１のカメラは、旋回体４の前方かつ中位の高さを撮影し、第２のカメラは、旋回体４の前方かつ上位の高さを撮影し、第３のカメラは、旋回体４の前方かつ下位の高さを撮影する。第１～第３のカメラは、作業装置５及びその近傍の映像を撮影する。第４のカメラは、旋回体４の左側かつ前方を撮影し、作業装置５から左側に離れた所定の場所の映像を撮影する。第５のカメラは、旋回体４の右側かつ前方を撮影し、作業装置５から右側に離れた所定の場所の映像を撮影する。なお、撮影装置１３は、例えば１つの広角カメラで構成されてもよい。 The photographing device 13 is composed of five cameras in this embodiment, but illustration of some of them is omitted. The first camera photographs the front and middle height of the rotating body 4, the second camera photographs the front and upper height of the rotating body 4, and the third camera photographs the front and middle height of the rotating body 4. Photograph the front and lower height of the The first to third cameras take images of the work device 5 and its vicinity. The fourth camera photographs the left side and front of the revolving structure 4, and photographs a predetermined location away from the working device 5 to the left. The fifth camera photographs the right side and the front of the rotating body 4, and photographs an image of a predetermined location away from the working device 5 to the right. Note that the photographing device 13 may be configured with, for example, one wide-angle camera.
　制御装置１６は、プログラムに従って処理を実行するプロセッサと、プログラムやデータを記憶するメモリ等を有する。制御装置１６は、機能的構成として、操作信号受信部１７、駆動制御部１８、通信品質受信部１９、動作認識部２０、圧縮設定部２１、映像データ圧縮部２２、及び映像データ送信部２３を有する。 The control device 16 has a processor that executes processing according to a program, a memory that stores programs and data, and the like. The control device 16 has a functional configuration including an operation signal receiving section 17, a drive control section 18, a communication quality receiving section 19, a motion recognition section 20, a compression setting section 21, a video data compression section 22, and a video data transmission section 23. have
　制御装置１６の操作信号受信部１７は、通信装置１５を介し遠隔操作装置２からの操作信号（詳細には、走行体３の走行、旋回体４の旋回、ブーム６の回動、アーム７の回動、又はバケット８の回動を指示する操作信号）を受信する。制御装置１６の駆動制御部１８は、操作信号受信部１７で受信された操作信号に応じて駆動装置１２（詳細には、対応する電磁比例弁）を制御する。これにより、走行体３の走行、旋回体４の旋回、ブーム６の回動、アーム７の回動、又はバケット８の回動を行わせる。 The operation signal receiving unit 17 of the control device 16 receives operation signals from the remote control device 2 via the communication device 15 (in detail, the running of the traveling body 3, the rotation of the rotating body 4, the rotation of the boom 6, and the rotation of the arm 7). or an operation signal instructing the rotation of the bucket 8). The drive control section 18 of the control device 16 controls the drive device 12 (specifically, the corresponding electromagnetic proportional valve) according to the operation signal received by the operation signal reception section 17. This causes the traveling body 3 to travel, the revolving body 4 to rotate, the boom 6 to rotate, the arm 7 to rotate, or the bucket 8 to rotate.
　制御装置１６の通信品質受信部１９は、通信装置１５を介し遠隔操作装置２からの通信品質の評価値（評価結果）を受信する。制御装置１６の動作認識部２０は、複数の角度センサ１４の検出結果に基づき、作業装置５の動作（詳細には、ブーム６、アーム７、及びバケット８の位置やバケット８の移動方向）及び旋回体４の動作（詳細には、旋回体４の旋回方向）を認識する。制御装置１６の圧縮設定部２１は、通信品質受信部１９で受信された通信品質の評価値と動作認識部２０で認識された作業装置５又は旋回体４の動作に基づいて、撮影装置１３で撮影された映像を示すデータ（以下、映像データと称する）の圧縮率などを設定する（詳細は後述）。 The communication quality receiving unit 19 of the control device 16 receives the communication quality evaluation value (evaluation result) from the remote control device 2 via the communication device 15. The motion recognition unit 20 of the control device 16 recognizes the motion of the working device 5 (specifically, the positions of the boom 6, arm 7, and bucket 8, and the moving direction of the bucket 8) based on the detection results of the plurality of angle sensors 14. The operation of the rotating body 4 (specifically, the turning direction of the rotating body 4) is recognized. The compression setting section 21 of the control device 16 sets the compression setting section 21 in the photographing device 13 based on the communication quality evaluation value received by the communication quality receiving section 19 and the motion of the working device 5 or the rotating body 4 recognized by the motion recognition section 20. The compression ratio of data indicating the photographed video (hereinafter referred to as video data) is set (details will be described later).
　制御装置１６の映像データ圧縮部２２は、圧縮設定部２１で設定された圧縮率などに基づいて、映像データを圧縮することにより、送信用の映像データを生成する。制御装置１６の映像データ送信部２３は、映像データ圧縮部２２で生成された送信用の映像データを、通信装置１５を介し遠隔操作装置２へ送信する。 The video data compression unit 22 of the control device 16 generates video data for transmission by compressing the video data based on the compression rate set by the compression setting unit 21. The video data transmitter 23 of the control device 16 transmits the video data for transmission generated by the video data compressor 22 to the remote control device 2 via the communication device 15.
　遠隔操作装置２は、映像を含め、種々の情報を表示する表示装置２４を備える。表示装置２４は、例えば、ショベル１の撮影装置１３を構成する５つのカメラにそれぞれ対応し、５つのカメラで撮影された映像をそれぞれ表示する５つの表示画面２５Ａ，２５Ｂ，２５Ｃ，２５Ｄ，２５Ｅを有する。 The remote control device 2 includes a display device 24 that displays various information including images. The display device 24 has, for example, five display screens 25A, 25B, 25C, 25D, and 25E that respectively correspond to the five cameras that make up the photographing device 13 of the excavator 1 and display images taken by the five cameras. have
　遠隔操作装置２は、操作者の操作に応じて操作信号を生成する操作装置２６を備える。操作装置２６は、操作者が操作可能な複数の操作部材（例えば操作レバー及び操作ペダル）と、複数の操作部材の操作方向及び操作量に対応する操作信号を生成する複数のポテンショメータとを有する。これにより、ショベル１の走行体３の走行、旋回体４の旋回、ブーム６の回動、アーム７の回動、又はバケット８の回動を指示する操作信号を生成する。 The remote control device 2 includes a control device 26 that generates a control signal in response to an operation by an operator. The operating device 26 includes a plurality of operating members (for example, an operating lever and an operating pedal) that can be operated by an operator, and a plurality of potentiometers that generate operating signals corresponding to the operating directions and amounts of the operating members. As a result, an operation signal is generated that instructs the traveling body 3 of the excavator 1 to travel, the rotating body 4 to rotate, the boom 6 to rotate, the arm 7 to rotate, or the bucket 8 to rotate.
　遠隔操作装置２は、ショベル１との通信を行うための通信装置２７と、制御装置２８とを備える。制御装置２８は、プログラムに従って処理を実行するプロセッサと、プログラムやデータを記憶するメモリ等を有する。制御装置２８は、機能的構成として、映像データ受信部２９、表示制御部３０、通信品質評価部３１、通信品質送信部３２、操作信号送信部３３を有する。 The remote control device 2 includes a communication device 27 for communicating with the shovel 1 and a control device 28. The control device 28 includes a processor that executes processing according to a program, a memory that stores programs and data, and the like. The control device 28 has a video data reception section 29, a display control section 30, a communication quality evaluation section 31, a communication quality transmission section 32, and an operation signal transmission section 33 as functional components.
　制御装置２８の映像データ受信部２９は、通信装置２７を介しショベル１からの映像データを受信する。制御装置２８の表示制御部３０は、映像データ受信部２９で受信された映像データに対応する映像を表示装置２４に表示させる。 The video data receiving unit 29 of the control device 28 receives video data from the shovel 1 via the communication device 27. The display control unit 30 of the control device 28 causes the display device 24 to display a video corresponding to the video data received by the video data receiving unit 29.
　本実施形態においては、遠隔操作装置２における制御装置２８の通信品質評価部３１は、映像データ受信部２９で受信された映像データに基づいて、ショベル１と遠隔操作装置２との間の通信の品質を評価する。詳しく説明すると、通信品質評価部３１は、例えば映像データの送信周期（固定値）と受信周期（変動値）との差により、映像データの通信にかかる遅延時間を算出する。そして、映像データの通信遅延時間やパケットロスに基づいて、通信品質の評価値（詳細には、通信品質が低いほど、小さくなる数値）を算出する。制御装置２８の通信品質送信部３２は、通信品質評価部３１で算出された通信品質の評価値を、通信装置２７を介しショベル１へ送信する。 In this embodiment, the communication quality evaluation section 31 of the control device 28 in the remote control device 2 determines the communication quality between the excavator 1 and the remote control device 2 based on the video data received by the video data receiving section 29. Evaluate quality. To explain in detail, the communication quality evaluation unit 31 calculates the delay time required for communication of video data, for example, based on the difference between the transmission cycle (fixed value) and the reception cycle (variable value) of the video data. Then, an evaluation value of communication quality (specifically, the lower the communication quality, the smaller the value) is calculated based on the communication delay time and packet loss of the video data. The communication quality transmitting unit 32 of the control device 28 transmits the communication quality evaluation value calculated by the communication quality evaluation unit 31 to the excavator 1 via the communication device 27.
　次に、ショベル１の制御装置１６の映像データの圧縮処理の詳細について、図４を用いて説明する。図４は、本実施形態におけるショベルの制御装置の映像データ圧縮処理の手順を表すフローチャートである。 Next, details of the video data compression process of the control device 16 of the shovel 1 will be explained using FIG. 4. FIG. 4 is a flowchart showing the procedure of video data compression processing of the excavator control device in this embodiment.
　ステップＳ１にて、制御装置１６の通信品質受信部１９は、通信装置１５を介し遠隔操作装置２からの通信品質の評価値を受信する。ステップＳ２にて、制御装置１６の圧縮設定部２１は、通信品質受信部１９で受信された通信品質の評価値が予め設定された閾値未満であるかどうかを判定する。 In step S1, the communication quality receiving unit 19 of the control device 16 receives the communication quality evaluation value from the remote control device 2 via the communication device 15. In step S2, the compression setting unit 21 of the control device 16 determines whether the communication quality evaluation value received by the communication quality receiving unit 19 is less than a preset threshold.
　通信品質の評価値が閾値以上である場合（言い換えれば、通信品質が低下していない場合）、ステップＳ３に移る。ステップＳ３にて、制御装置１６の圧縮設定部２１は、映像データの全体のデータ量が標準値になるように、標準の圧縮率を設定する。また、制御装置１６の映像データ圧縮部２２は、標準の圧縮率にて、撮影装置１３で撮影された映像（本実施形態では、５つのカメラで撮影された映像Ａ１～Ａ５）を示す映像データの全体に対して均一に圧縮して送信用の映像データを生成する。そして、映像データ送信部２３は、当該圧縮された送信用の映像データを送信する。 If the communication quality evaluation value is equal to or greater than the threshold (in other words, if the communication quality has not deteriorated), the process moves to step S3. In step S3, the compression setting unit 21 of the control device 16 sets a standard compression rate so that the total amount of video data becomes a standard value. In addition, the video data compression unit 22 of the control device 16 compresses video data representing videos shot by the imaging device 13 (in this embodiment, videos A1 to A5 shot by five cameras) at a standard compression rate. The entire video data is uniformly compressed to generate video data for transmission. Then, the video data transmitter 23 transmits the compressed video data for transmission.
　一方、通信品質の評価値が閾値未満である場合（言い換えれば、通信品質が低下している場合）、ステップＳ４に移る。ステップＳ４にて、制御装置１６の圧縮設定部２１は、通信品質の評価値が低くなるほど、映像データの全体のデータ低減量（詳細には、上記標準値から減らす量）を多くするように設定する。すなわち、圧縮設定部２１は、通信品質の評価値が低くなるほど、映像データの圧縮率を大きくするように設定する。また、ステップＳ５にて、動作認識部２０は、作業装置５の動作又は旋回体４の動作が認識されたかどうかを判定する。 On the other hand, if the communication quality evaluation value is less than the threshold (in other words, if the communication quality is decreasing), the process moves to step S4. In step S4, the compression setting unit 21 of the control device 16 sets the overall data reduction amount of the video data (specifically, the amount of reduction from the standard value) to increase as the evaluation value of the communication quality becomes lower. do. That is, the compression setting unit 21 sets the compression ratio of the video data to be increased as the communication quality evaluation value becomes lower. Further, in step S5, the motion recognition unit 20 determines whether the motion of the working device 5 or the motion of the rotating body 4 has been recognized.
　作業装置５の動作及び旋回体４の動作が認識されない場合、ステップＳ３に移る。ステップＳ３にて、制御装置１６の映像データ圧縮部２２は、ステップＳ４で設定された圧縮率にて、撮影装置１３で撮影された映像の全体を均一に圧縮し、映像データ送信部２３は、圧縮された映像を送信する。 If the operation of the working device 5 and the operation of the revolving structure 4 are not recognized, the process moves to step S3. In step S3, the video data compression unit 22 of the control device 16 uniformly compresses the entire video shot by the imaging device 13 at the compression rate set in step S4, and the video data transmission unit 23 Send compressed video.
　一方、ステップＳ５において作業装置５の動作又は旋回体４の動作が認識された場合、ステップＳ６に移る。ステップＳ６にて、制御装置１６の圧縮設定部２１は、映像を複数の映像領域に分割する（区画する）と共に、複数の映像領域に対応する複数の映像データ（以下、映像領域データと称する）のそれぞれに対し、データ低減の優先順位を設定する。その後、ステップＳ７にて、圧縮設定部２１は、ステップＳ４で設定された映像データの全体のデータ低減量となるように、ステップＳ６で設定されたデータ低減の優先順位に基づいて、複数の映像領域データのデータ低減量又は低減率をそれぞれ設定する。すなわち、圧縮設定部２１は、作業装置５の動作状況及び遠隔操作装置２との通信状況に基づいて、表示装置２４に表示させる映像データのデータ低減量（圧縮率）を、映像の映像領域単位で個別に設定する。 On the other hand, if the operation of the working device 5 or the operation of the rotating body 4 is recognized in step S5, the process moves to step S6. In step S6, the compression setting unit 21 of the control device 16 divides (divides) the video into a plurality of video regions, and also creates a plurality of video data (hereinafter referred to as video region data) corresponding to the plurality of video regions. Set a data reduction priority for each. Thereafter, in step S7, the compression setting unit 21 selects a plurality of video data based on the data reduction priority set in step S6 so as to achieve the total data reduction amount of the video data set in step S4. Set the data reduction amount or reduction rate of the area data. That is, the compression setting unit 21 determines the amount of data reduction (compression rate) of the video data to be displayed on the display device 24 based on the operating status of the work device 5 and the communication status with the remote control device 2 in units of video areas of the video. Set individually.
　詳しく説明すると、制御装置１６の圧縮設定部２１は、今回又は前回の作業装置５の動作の認識により、５つのカメラで撮影された映像Ａ１～Ａ５に対応する映像データのうち、作業装置５のバケット８が現れる映像データを選択する。例えば図５で示すように、選択された映像Ａ１に対応する映像データを、作業装置５のバケット８及びその周囲を示す主の映像領域Ｒ１に対応する主映像領域データと、作業装置５のブーム６を示す映像領域Ｒ２に対応する映像領域データと、他の映像領域Ｒ３に対応する副映像領域データと、に分割する。 To explain in detail, the compression setting unit 21 of the control device 16 recognizes the current or previous operation of the working device 5 and selects the image data of the working device 5 from among the video data corresponding to the images A1 to A5 shot by the five cameras. Select the video data in which bucket 8 appears. For example, as shown in FIG. 5, the video data corresponding to the selected video A1 is combined with the main video area data corresponding to the main video area R1 showing the bucket 8 of the work device 5 and its surroundings, and the boom of the work device 5. The video area data is divided into video area data corresponding to the video area R2 indicating 6 and sub-video area data corresponding to another video area R3.
　また、残りの映像Ａ２～Ａ５に対応する映像データについては、例えば映像Ａ１に対応する映像データと同様である。すなわち、例えば、作業装置５のブーム６が現れていなければ、映像Ａ１の映像領域Ｒ３に対応する映像領域データと同様の扱いとする。そして、データ低減の優先順位の１番目は映像Ａ１の映像領域Ｒ２に対応する映像領域データ、２番目は映像Ａ１の映像領域Ｒ３に対応する映像領域データ及び映像Ａ２～Ａ５に対応する映像領域データ、３番目は映像Ａ１の映像領域Ｒ１に対応する映像領域データと設定する。そして、ステップＳ４で設定された映像データの全体のデータ低減量となるように、前述したデータ低減の優先順位に基づいて、映像Ａ１の映像領域Ｒ１，Ｒ２，Ｒ３及び映像Ａ２～Ａ５に対応する映像領域データのデータ低減量をそれぞれ設定する。 Furthermore, the video data corresponding to the remaining videos A2 to A5 is the same as, for example, the video data corresponding to video A1. That is, for example, if the boom 6 of the working device 5 does not appear, the processing is the same as the video area data corresponding to the video area R3 of the video A1. The first data reduction priority is video area data corresponding to video area R2 of video A1, and the second is video area data corresponding to video area R3 of video A1 and video area data corresponding to videos A2 to A5. , the third is set as video area data corresponding to video area R1 of video A1. Then, based on the data reduction priority order described above, corresponding to video regions R1, R2, R3 of video A1 and videos A2 to A5, so as to achieve the data reduction amount of the entire video data set in step S4. Set the amount of data reduction for each video area data.
　そして、ステップＳ７にて、制御装置１６の映像データ圧縮部２２は、ステップＳ６で設定された映像領域ごとの映像データである映像領域データのデータ低減量にて、撮影装置１３で撮影された映像を不均一に圧縮した送信用の映像データを生成する。映像データ送信部２３は、当該圧縮された送信用の映像データを送信する。 Then, in step S7, the video data compression unit 22 of the control device 16 reduces the amount of data of the video region data, which is the video data for each video region set in step S6, to reduce the amount of video captured by the imaging device 13. Generates video data for transmission that is non-uniformly compressed. The video data transmitter 23 transmits the compressed video data for transmission.
　以上のように本実施形態においては、ショベル１と遠隔操作装置２の間の通信の品質が低下したときに、映像データの全体のデータ量を低減する。これにより、通信の品質の低下を抑えることができる。そのため、作業状況の把握に直結する映像の表示遅延を防止し、これに伴う操作装置２６の操作の遅れを抑えて、ショベル１の生産性の低下を抑えることができる。 As described above, in this embodiment, when the quality of communication between the shovel 1 and the remote control device 2 deteriorates, the total amount of video data is reduced. Thereby, deterioration in communication quality can be suppressed. Therefore, it is possible to prevent a delay in displaying images that are directly linked to grasping the work situation, to suppress a delay in operating the operating device 26 due to this, and to suppress a decrease in productivity of the shovel 1.
　また、本実施形態においては、作業装置５の動作又は旋回体４の動作に応じて、作業装置５のバケット８及びその周囲を示す主の映像領域Ｒ１と他の映像領域を含む複数の映像領域に映像を分割すると共に、主の映像領域Ｒ１に対応する映像領域データである主映像領域データと、他の映像領域に対応する副映像領域データとを生成する。そして、主映像領域データより副映像領域データを優先するようにデータ削減の優先順位を設定する。例えば、主映像領域データより副映像領域データの低減量又は低減率を大きくするか、若しくは主映像領域データのデータ量より副映像領域データのデータ量方が大きくなるようにデータ低減を行う。これにより、表示装置２４で表示された映像のうち、主の映像領域Ｒ１の解像度の低下を抑える。すなわち、ショベル１の動作に必要な領域を自動的に抽出して、その解像度の低下を抑えることができる。したがって、ショベル１の生産性の低下を抑えることができる。 In addition, in the present embodiment, a plurality of video regions including a main video region R1 showing the bucket 8 of the work device 5 and its surroundings and other video regions are created according to the operation of the work device 5 or the operation of the revolving structure 4. At the same time, main video area data that is video area data corresponding to the main video area R1 and sub-video area data corresponding to other video areas are generated. Then, the data reduction priority order is set so that the sub-video area data is prioritized over the main video area data. For example, data reduction is performed such that the reduction amount or reduction rate of the sub-video area data is greater than that of the main video area data, or the data amount of the sub-video area data is larger than the data amount of the main video area data. This suppresses a decrease in the resolution of the main video region R1 of the video displayed on the display device 24. That is, it is possible to automatically extract the area necessary for the operation of the shovel 1 and suppress a decrease in its resolution. Therefore, a decrease in productivity of the shovel 1 can be suppressed.
　なお、第１の実施形態において、ショベル１の制御装置１６は、図５で示すように、バケット８の注視すべき部分（詳細には、バケット８の開口及び爪）の中心を楕円形の中心点として、主の映像領域Ｒ１を楕円形の領域として設定する場合を例にとって説明したが、これに限られない。制御装置１６は、図６Ａ又は図６Ｂで示すように、バケット８の注視すべき部分の中心を楕円形の焦点として、主の映像領域Ｒ１を楕円形の領域に設定してもよい。すなわち、図６Ａで示すように、バケット８が下側に移動するとき、バケット８の領域を基準としてその移動方向側の領域がその反対方向側の領域より大きくなるように主の映像領域Ｒ１の形状および範囲を設定してもよい。また、図６Ｂで示すように、バケット８が左側に移動するとき、バケット８の領域を基準としてその左側の領域が右側の領域より大きくなるような形状に主の映像領域Ｒ１を設定してもよい。 In the first embodiment, as shown in FIG. 5, the control device 16 of the shovel 1 moves the center of the part of the bucket 8 to be watched (specifically, the opening and claw of the bucket 8) to the center of the ellipse. Although the explanation has been given using an example in which the main video region R1 is set as an elliptical region, the present invention is not limited to this. As shown in FIG. 6A or 6B, the control device 16 may set the main image region R1 to be an elliptical region with the center of the portion of the bucket 8 to be watched as the focal point of the ellipse. That is, as shown in FIG. 6A, when the bucket 8 moves downward, the main video area R1 is adjusted so that the area on the side of the moving direction is larger than the area on the opposite side with respect to the area of the bucket 8. The shape and range may be set. Furthermore, as shown in FIG. 6B, when the bucket 8 moves to the left, the main video area R1 may be set in such a shape that the area on the left is larger than the area on the right based on the area of the bucket 8. good.
　また、第１の実施形態において、ショベル１の制御装置１６は、旋回体４の動作が認識されたときに、作業装置５の動作が認識されたときと同様、作業装置５のバケット８及びその周囲を示す主の映像領域Ｒ１と他の映像領域を含む複数の映像領域に映像を分割し、それぞれの映像領域に対応する映像領域データごとに（例えば主映像領域データ及び副映像領域データとでデータ低減率を変えて）圧縮処理を行う場合を例にとって説明したが、これに限られない。 In addition, in the first embodiment, when the operation of the rotating structure 4 is recognized, the control device 16 of the excavator 1 controls the bucket 8 of the working device 5 and its The video is divided into a plurality of video regions including a main video region R1 indicating the surroundings and other video regions, and each video region data corresponding to each video region is divided into two (for example, main video region data and sub video region data). Although the case where the compression process is performed (by changing the data reduction rate) has been described as an example, the present invention is not limited to this.
　例えば、制御装置１６は、旋回体４の動作が認識されたときに、作業装置５の移動先を示す主の映像領域と他の映像領域を含む複数の映像領域に映像を分割し、このように分割された映像領域に対応する映像領域データ（それぞれ主映像領域データ及び副映像領域データ）に対して圧縮処理を行ってもよい。すなわち、例えば旋回体４の左旋回に伴って作業装置５が左側に移動するとき、５つのカメラで撮影された映像Ａ１～Ａ５に対応する映像データを、映像Ａ４（主の映像領域）に対応する主映像領域データと、他の映像Ａ１－Ａ３，Ａ５（他の映像領域）に対応する副映像領域データとに分割し、映像Ａ４に対応する主映像領域データより映像Ａ１－Ａ３，Ａ５に対応する副映像領域データを優先して圧縮するようにデータ削減の優先順位を設定してもよい。また、例えば旋回体４の右旋回に伴って作業装置５が右側に移動するとき、５つのカメラで撮影された映像Ａ１～Ａ５に対応する映像データを、映像Ａ５（主の映像領域）に対応する主映像領域データと、他の映像Ａ１－Ａ４（他の映像領域）に対応する副映像領域データとに分割し、映像Ａ５に対応する主映像領域データより映像Ａ１－Ａ４に対応する副映像領域データを優先するようにデータ削減の優先順位を設定してもよい。 For example, when the motion of the rotating body 4 is recognized, the control device 16 divides the image into a plurality of image areas including a main image area indicating the movement destination of the working device 5 and other image areas, and Compression processing may be performed on video area data (main video area data and sub-video area data, respectively) corresponding to the divided video areas. That is, for example, when the working device 5 moves to the left as the rotating body 4 turns to the left, video data corresponding to videos A1 to A5 taken by five cameras is transferred to video A4 (main video area). The main video area data corresponding to video A4 is divided into main video area data corresponding to other video A1-A3, A5 (other video areas), and the main video area data corresponding to video A4 is divided into video A1-A3, A5. Data reduction priorities may be set so that corresponding sub-video area data is preferentially compressed. For example, when the working device 5 moves to the right as the revolving structure 4 turns to the right, video data corresponding to videos A1 to A5 taken by five cameras is transferred to video A5 (main video area). The main video area data corresponding to the video A1-A4 (other video areas) are divided into the corresponding main video area data and the sub video area data corresponding to the other videos A1-A4 (other video areas), and the sub video area data corresponding to the videos A1-A4 is divided from the main video area data corresponding to the video A5. The priority order of data reduction may be set to give priority to video area data.
　本発明の第２の実施形態を、図７を用いて説明する。図７は、本実施形態におけるショベルの制御装置の機能的構成を関連機器と共に表すブロック図である。なお、本実施形態において、第１の実施形態と同等の部分は同一の符号を付し、適宜、説明を省略する。 A second embodiment of the present invention will be described using FIG. 7. FIG. 7 is a block diagram showing the functional configuration of the excavator control device in this embodiment together with related equipment. In addition, in this embodiment, parts equivalent to those in the first embodiment are given the same reference numerals, and description thereof will be omitted as appropriate.
　本実施形態のショベル１の制御装置１６Ａは、第１の実施形態の制御装置１６と同様、操作信号受信部１７、駆動制御部１８、通信品質受信部１９、動作認識部２０、圧縮設定部２１、映像データ圧縮部２２、及び映像データ送信部２３を有する。 Similar to the control device 16 of the first embodiment, the control device 16A of the excavator 1 of this embodiment includes an operation signal receiving section 17, a drive control section 18, a communication quality receiving section 19, a motion recognition section 20, and a compression setting section 21. , a video data compression section 22, and a video data transmission section 23.
　制御装置１６Ａは、移動体検知部３４を更に有する。制御装置１６Ａの移動体検知部３４は、撮影装置１３で撮影された映像の処理（若しくは、移動体検知センサ３５の検知）により、ショベル１の周囲に存在するダンプトラック（移動体）を検知する。 The control device 16A further includes a moving object detection section 34. The moving object detection unit 34 of the control device 16A detects a dump truck (moving object) existing around the excavator 1 by processing the image photographed by the photographing device 13 (or by detecting the moving object detection sensor 35). .
　制御装置１６Ａの圧縮設定部２１は、通信品質の評価値が閾値未満である場合、映像データの全体のデータ低減量を設定する。そして、移動体検知部３４でダンプトラックが検知された場合、５つのカメラで撮影された映像Ａ１～Ａ５に対応する映像データのうち、ダンプトラックが現れる映像データを選択する。例えば、選択された映像Ａ４に対応する映像データを、ダンプトラック及びその周囲を示す主の映像領域に対応する主映像領域データと、他の映像領域に対応する副映像領域データと、に分割する。 The compression setting unit 21 of the control device 16A sets the overall data reduction amount of the video data when the communication quality evaluation value is less than the threshold value. When a dump truck is detected by the moving object detection unit 34, the video data in which the dump truck appears is selected from among the video data corresponding to the videos A1 to A5 taken by the five cameras. For example, the video data corresponding to the selected video A4 is divided into main video area data corresponding to the main video area showing the dump truck and its surroundings, and sub-video area data corresponding to other video areas. .
　また、残りの映像Ａ１～Ａ３，Ａ５に対応する映像データについては、例えば映像Ａ４に対応する映像データと同様である。すなわち、例えば、ダンプトラックが現れていなければ、映像Ａ４の他の映像領域に対応する映像領域データと同様の扱いとする。そして、データ低減の優先順位の１番目は映像Ａ４の他の映像領域に対応する映像領域データ及び映像Ａ１～Ａ３，Ａ５に対応する映像領域データ、２番目は映像Ａ４の主の映像領域に対応する映像領域データと設定する。そして、設定された映像データの全体のデータ低減量となるように、前述したデータ低減の優先順位に基づいて、映像Ａ４の主の映像領域、他の映像領域，及び映像Ａ１～Ａ３，Ａ５に対応する映像領域データのデータ低減量をそれぞれ設定する。 Furthermore, the video data corresponding to the remaining videos A1 to A3 and A5 is the same as, for example, the video data corresponding to video A4. That is, for example, if a dump truck does not appear, the data is handled in the same way as video area data corresponding to other video areas of video A4. The first data reduction priority is the video area data corresponding to other video areas of video A4 and the video area data corresponding to videos A1 to A3, and A5, and the second is the video area data corresponding to the main video area of video A4. Set the video area data to be used. Then, based on the data reduction priority described above, the main video area of video A4, other video areas, and videos A1 to A3 and A5 are Set the amount of data reduction for the corresponding video area data.
　以上のように構成された本実施形態においても、第１の実施形態と同様、ショベル１と遠隔操作装置２の間の通信の品質の低下を抑えつつ、ショベル１の生産性の低下を抑えることができる。 Also in the present embodiment configured as described above, similarly to the first embodiment, it is possible to suppress a decrease in the productivity of the excavator 1 while suppressing a decrease in the quality of communication between the excavator 1 and the remote control device 2. I can do it.
　本発明の第３の実施形態を、図８を用いて説明する。図８は、本実施形態におけるショベルの制御装置の機能的構成を関連機器と共に表すブロック図である。なお、本実施形態において、第１及び第２の実施形態と同等の部分は同一の符号を付し、適宜、説明を省略する。 A third embodiment of the present invention will be described using FIG. 8. FIG. 8 is a block diagram showing the functional configuration of the excavator control device in this embodiment together with related equipment. In this embodiment, parts equivalent to those in the first and second embodiments are denoted by the same reference numerals, and description thereof will be omitted as appropriate.
　本実施形態のショベル１の制御装置１６Ｂは、第２の実施形態の制御装置１６Ａと同様、操作信号受信部１７、駆動制御部１８、通信品質受信部１９、圧縮設定部２１、映像データ圧縮部２２、映像データ送信部２３、及び移動体検知部３４を有する。 The control device 16B of the excavator 1 of this embodiment, like the control device 16A of the second embodiment, includes an operation signal receiving section 17, a drive control section 18, a communication quality receiving section 19, a compression setting section 21, and a video data compression section. 22, a video data transmitter 23, and a moving object detector 34.
　制御装置１６Ｂは、第２の実施形態の制御装置１６Ａの動作認識部２０に代えて、動作予測部３６を有する。制御装置１６Ｂの動作予測部３６は、動作認識部２０と同様、複数の角度センサの検出結果から作業装置５の動作及び旋回体４の動作を認識する。そして、作業装置５の動作又は旋回体４の動作の推移や、移動体検知部３４によるダンプトラックの検知の有無により、ショベル１の各作業に係わるショベルの動作パターンを学習して記憶する。そして、前述した動作パターンを用いて、現在のショベルの動作若しくは現在の車両の検知から、将来のショベルの動作を予測する。 The control device 16B has a motion prediction section 36 in place of the motion recognition section 20 of the control device 16A of the second embodiment. The motion prediction section 36 of the control device 16B, like the motion recognition section 20, recognizes the motion of the working device 5 and the motion of the rotating structure 4 from the detection results of the plurality of angle sensors. Then, the operating pattern of the excavator 1 related to each work of the excavator 1 is learned and stored based on the transition of the operation of the working device 5 or the operation of the revolving structure 4, and whether or not a dump truck is detected by the moving object detection unit 34. Then, using the above-mentioned operation pattern, the future operation of the excavator is predicted from the current operation of the excavator or the current detection of the vehicle.
　具体例として、ショベル１の積込作業に係わるショベルの動作パターンを用いて説明する。この動作パターンでは、まず、例えばショベル１の作業装置５から左側又は右側に離れた所定の場所に、ダンプトラックが現れる。その後、ショベル１の旋回体４がダンプトラックに向かうように旋回する。その後、ショベル１の作業装置５が動作する。 As a specific example, an explanation will be given using an operation pattern of the shovel related to the loading work of the shovel 1. In this operation pattern, first, a dump truck appears at a predetermined location away from, for example, the left or right side of the working device 5 of the shovel 1. Thereafter, the revolving body 4 of the excavator 1 turns toward the dump truck. After that, the working device 5 of the shovel 1 operates.
　制御装置１６Ｂの動作予測部３６は、移動体検知部３４でダンプトラックが検知された場合、ショベル１の旋回体４がダンプトラックに向かうように旋回することを予測する。制御装置１６Ｂの圧縮設定部２１は、５つのカメラで撮影された映像Ａ１～Ａ５に対応する映像データのうち、ダンプトラックが現れる映像データを選択する。例えば、選択された映像Ａ４に対応する映像データを、ダンプトラック及びその周囲を示す主の映像領域に対応する主映像領域データと、他の映像領域に対応する副映像領域データと、に分割する。また、５つのカメラで撮影された映像Ａ１～Ａ５に対応する映像データのうち、作業装置５のバケット８が現れる映像データを選択する。例えば、選択された映像Ａ１に対応する映像データを、作業装置５のバケット８及びその周囲を示す主の映像領域Ｒ１に対応する主映像領域データと、作業装置５のブーム６を示す映像領域Ｒ２に対応する映像領域データと、他の映像領域Ｒ３に対応する副映像領域データと、に分割する（上述の図５又は図６Ｂ参照）。 The operation prediction unit 36 of the control device 16B predicts that the revolving body 4 of the excavator 1 will turn toward the dump truck when the moving object detection unit 34 detects a dump truck. The compression setting unit 21 of the control device 16B selects the video data in which the dump truck appears from among the video data corresponding to the videos A1 to A5 shot by the five cameras. For example, the video data corresponding to the selected video A4 is divided into main video area data corresponding to the main video area showing the dump truck and its surroundings, and sub-video area data corresponding to other video areas. . Furthermore, the video data in which the bucket 8 of the working device 5 appears is selected from among the video data corresponding to the videos A1 to A5 taken by the five cameras. For example, the video data corresponding to the selected video A1 is combined with the main video area data corresponding to the main video area R1 showing the bucket 8 of the work equipment 5 and its surroundings, and the video area R2 showing the boom 6 of the work equipment 5. and sub-video area data corresponding to another video area R3 (see FIG. 5 or FIG. 6B described above).
　また、残りの映像Ａ２，Ａ３，Ａ５に対応する映像データについては、例えば映像Ａ１に対応する映像データと同様である。すなわち、例えば、作業装置５のブーム６が現れていなければ、映像Ａ１の映像領域Ｒ３に対応する映像領域データと同様の扱いとする。そして、データ低減の優先順位の１番目は映像Ａ１の映像領域Ｒ２に対応する映像領域データ、２番目は映像Ａ１の映像領域Ｒ３に対応する映像領域データ、映像Ａ４の他の映像領域に対応する映像領域データ、及び映像Ａ２，Ａ３，Ａ５に対応する映像領域データ、３番目は映像Ａ１の映像領域Ｒ１に対応する映像領域データ及び映像Ａ４の主の映像領域に対応する映像領域データと設定する。そして、通信品質の低下に応じて設定された映像データの全体のデータ低減量となるように、前述したデータ低減の優先順位に基づいて、映像Ａ１の映像領域Ｒ１，Ｒ２，Ｒ３、映像Ａ４の主の映像領域及び他の映像領域、並びに映像Ａ２，Ａ３，Ａ５に対応する映像領域データのデータ低減量をそれぞれ設定する。 Furthermore, the video data corresponding to the remaining videos A2, A3, and A5 is the same as, for example, the video data corresponding to video A1. That is, for example, if the boom 6 of the working device 5 does not appear, the processing is the same as the video area data corresponding to the video area R3 of the video A1. The first data reduction priority is video area data corresponding to video area R2 of video A1, the second is video area data corresponding to video area R3 of video A1, and the second is video area data corresponding to other video areas of video A4. The third set is video area data corresponding to video area R1 of video A1 and video area data corresponding to the main video area of video A4. . Then, based on the data reduction priority mentioned above, video areas R1, R2, R3 of video A1, video areas R2, and R3 of video A4 are The amount of data reduction for video area data corresponding to the main video area, other video areas, and videos A2, A3, and A5 is set, respectively.
　以上のように構成された本実施形態においても、第１の実施形態と同様、ショベル１と遠隔操作装置２の間の通信の品質の低下を抑えつつ、ショベル１の生産性の低下を抑えることができる。 Also in the present embodiment configured as described above, similarly to the first embodiment, it is possible to suppress a decrease in the productivity of the excavator 1 while suppressing a decrease in the quality of communication between the excavator 1 and the remote control device 2. I can do it.
　なお、第１～第３の実施形態において、ショベル１は、作業装置５の動作に係わる状態量を検出するセンサとして、ブーム６の回動角、アーム７の回動角、及びバケット８の回動角をそれぞれ検出する複数の角度センサ１４を備えた場合を例にとって説明したが、これに限られない。ショベル１は、例えば、ブーム６の傾斜角、アーム７の傾斜角、及びバケット８の傾斜角をそれぞれ検出する複数の慣性計測装置を備えてもよい。そして、複数の慣性計測装置の検出結果に基づき、作業装置５の動作を認識してもよい。 In the first to third embodiments, the excavator 1 uses the rotation angle of the boom 6, the rotation angle of the arm 7, and the rotation angle of the bucket 8 as sensors for detecting state quantities related to the operation of the working device 5. Although a case has been described using as an example a case where a plurality of angle sensors 14 each detecting a moving angle are provided, the present invention is not limited to this. The excavator 1 may include a plurality of inertial measurement devices that respectively detect the inclination angle of the boom 6, the inclination angle of the arm 7, and the inclination angle of the bucket 8, for example. Then, the operation of the work device 5 may be recognized based on the detection results of the plurality of inertial measurement devices.
　また、第１～第３の実施形態において、ショベル１の制御装置１６は、複数の角度センサ１４の検出結果に基づき、作業装置５の動作及び旋回体４の動作を認識する場合を例にとって説明したが、これに限られない。遠隔操作装置２から受信した操作信号に基づき、作業装置５の動作及び旋回体４の動作を認識してもよい。 In addition, in the first to third embodiments, an example will be explained in which the control device 16 of the excavator 1 recognizes the operation of the working device 5 and the operation of the rotating structure 4 based on the detection results of the plurality of angle sensors 14. However, it is not limited to this. Based on the operation signal received from the remote control device 2, the operation of the working device 5 and the operation of the revolving structure 4 may be recognized.
　なお、以上においては、作業機械が大型のショベルである場合を例にとって説明したが、これに限られない。例えば、中型又は小型のショベルであってもよいし、ショベル以外の他の作業機械（詳細には、クレーン車等）であってもよい。 In addition, although the case where the working machine is a large excavator has been explained above as an example, the present invention is not limited to this. For example, it may be a medium-sized or small-sized excavator, or it may be a working machine other than an excavator (specifically, a crane truck, etc.).
　第１の発明は、作業装置、前記作業装置及びその周囲の映像を撮影する撮影装置、並びに前記撮影装置で撮影された映像を映像データとして第１の通信装置を介して送信すると共に、前記第１の通信装置を介して受信した操作信号に応じて前記作業装置の動作を制御する第１の制御装置を備えた作業機械と、種々の情報を表示する表示装置、操作者の操作に応じて前記操作信号を生成する操作装置、並びに前記作業機械から第２の通信装置を介して受信した前記映像データに対応する映像を前記表示装置に表示させると共に、前記操作装置で生成された前記操作信号を前記第２の通信装置を介して前記作業機械へ送信する第２の制御装置を備えた遠隔操作装置と、を備えた遠隔操作型作業機械システムにおいて、前記遠隔操作装置の前記第２の制御装置は、前記作業機械との通信の品質を評価し、当該評価結果を前記作業機械に送信し、前記作業機械の前記第１の制御装置は、前記操作信号に基づいて、前記作業装置の動作状況を判定し、前記遠隔操作装置から、前記作業機械と前記遠隔操作装置の間の通信品質が低下していることを示す評価結果を受信し、かつ前記作業装置が動作していると判定した場合、前記映像データを前記映像の複数の映像領域に対応する複数の映像領域データに分割すると共に、前記複数の映像領域データのそれぞれに対するデータ低減量を設定し、当該設定された前記複数の映像領域データごとのデータ低減量に基づいて低減された送信用の映像データを生成し、当該送信用の映像データを前記遠隔操作装置に送信する。 A first invention provides a working device, a photographing device for photographing an image of the working device and its surroundings, and transmitting the video photographed by the photographing device as video data via a first communication device; a working machine including a first control device that controls the operation of the working device in response to an operation signal received via a first communication device; a display device that displays various information; an operating device that generates the operating signal; and displaying an image corresponding to the video data received from the work machine via a second communication device on the display device, and the operating signal generated by the operating device. a remote control device including a second control device that transmits the information to the work machine via the second communication device, wherein the second control of the remote control device The device evaluates the quality of communication with the working machine and transmits the evaluation result to the working machine, and the first control device of the working machine controls the operation of the working machine based on the operation signal. determines the situation, receives an evaluation result from the remote control device indicating that the communication quality between the work machine and the remote control device is degraded, and determines that the work device is operating; In this case, the video data is divided into a plurality of video region data corresponding to a plurality of video regions of the video, and a data reduction amount is set for each of the plurality of video region data, and the set Reduced video data for transmission is generated based on the amount of data reduction for each region data, and the video data for transmission is transmitted to the remote control device.
　第２の発明は、作業装置、前記作業装置及びその周囲の映像を撮影する撮影装置、並びに前記撮影装置で撮影された映像を映像データとして第１の通信装置を介して送信すると共に、前記第１の通信装置を介して受信した操作信号に応じて前記作業装置の動作を制御する第１の制御装置を備えた作業機械と、種々の情報を表示する表示装置、操作者の操作に応じて前記操作信号を生成する操作装置、並びに前記作業機械から第２の通信装置を介して受信した前記映像データに対応する映像を前記表示装置に表示させると共に、前記操作装置で生成された前記操作信号を前記第２の通信装置を介して前記作業機械へ送信する第２の制御装置を備えた遠隔操作装置と、を備えた遠隔操作型作業機械システムにおいて、前記遠隔操作装置の前記第２の制御装置は、前記作業機械との通信の品質を評価し、当該評価結果を前記作業機械に送信し、前記作業機械の前記第１の制御装置は、前記操作信号に基づいて、前記作業装置の動作状況を判定し、前記遠隔操作装置から、前記作業機械と前記遠隔操作装置の間の通信品質が低下していることを示す評価結果を受信し、かつ前記作業装置が動作していると判定した場合、前記映像データを前記映像の複数の映像領域に対応する複数の映像領域データに分割すると共に、前記複数の映像領域データのそれぞれに対するデータ低減量を設定し、当該設定された前記複数の映像領域データごとのデータ低減量に基づいて低減された送信用の映像データを生成し、当該送信用の映像データを前記遠隔操作装置に送信する。 A second invention includes a working device, a photographing device for photographing an image of the working device and its surroundings, and transmitting the video photographed by the photographing device as video data via a first communication device; a working machine including a first control device that controls the operation of the working device in response to an operation signal received via a first communication device; a display device that displays various information; an operating device that generates the operating signal; and displaying an image corresponding to the video data received from the work machine via a second communication device on the display device, and the operating signal generated by the operating device. a remote control device including a second control device that transmits the information to the work machine via the second communication device, wherein the second control of the remote control device The device evaluates the quality of communication with the working machine and transmits the evaluation result to the working machine, and the first control device of the working machine controls the operation of the working machine based on the operation signal. determines the situation, receives an evaluation result from the remote control device indicating that the communication quality between the work machine and the remote control device is degraded, and determines that the work device is operating; In this case, the video data is divided into a plurality of video region data corresponding to a plurality of video regions of the video, and a data reduction amount is set for each of the plurality of video region data, and the set Reduced video data for transmission is generated based on the amount of data reduction for each region data, and the video data for transmission is transmitted to the remote control device.
　上記第１又は第２の発明によれば、作業機械と遠隔操作装置との間の通信品質が低下した場合でも、例えば、遠隔操作装置に表示可能な映像領域のうちの作業に直結するような領域の映像品質を犠牲にすることなく、リアルタイムかつ高品質な映像を遠隔操作装置に表示させることができ、操作や作業の遅延、ひいては作業機械の生産性の低下を抑えることができる。 According to the first or second aspect of the invention, even if the communication quality between the work machine and the remote control device is degraded, for example, a video area that can be displayed on the remote control device that is directly connected to the work is Real-time, high-quality images can be displayed on a remote control device without sacrificing the image quality of the area, making it possible to suppress delays in operations and work and, ultimately, a decline in productivity of working machines.
　１　　　　　　　　　　　ショベル（作業機械）
　２　　　　　　　　　　　遠隔操作装置
　３　　　　　　　　　　　走行体
　４　　　　　　　　　　　旋回体
　５　　　　　　　　　　　作業装置
　８　　　　　　　　　　　バケット（作業具）
　１３　　　　　　　　　　撮影装置
　１５　　　　　　　　　　通信装置
　１６，１６Ａ，１６Ｂ　　制御装置
　２４　　　　　　　　　　表示装置
　２６　　　　　　　　　　操作装置
　２７　　　　　　　　　　通信装置
　２８　　　　　　　　　　制御装置
　３５　　　　　　　　　　移動体検知センサ
　１００　　　　　　　　　ショベルシステム（作業機械システム） 1 Excavator (work machine)
2 Remote control device 3 Traveling body 4 Swivel body 5 Working device 8 Bucket (working tool)
13 Photographing device 15 Communication device 16, 16A, 16B Control device 24 Display device 26 Operating device 27 Communication device 28 Control device 35 Moving object detection sensor 100 Shovel system (work machine system)

Claims (7)

  1. 　作業装置と、前記作業装置及びその周囲の映像を撮影する撮影装置と、前記撮影装置で撮影された映像を映像データとして通信装置を介して遠隔操作装置へ送信すると共に、前記遠隔操作装置から前記通信装置を介して受信した操作信号に応じて前記作業装置の動作を制御する制御装置と、を備えた作業機械において、
    　前記制御装置は、
    　前記遠隔操作装置から、前記通信装置を介して前記遠隔操作装置との間の通信品質の評価結果を受信し、
    　前記操作信号に基づいて、前記作業装置の動作状況を判定し、
    　前記通信品質の評価結果及び前記作業装置の動作状況に基づいて、前記映像データを前記映像の複数の映像領域に対応する複数の映像領域データに分割すると共に、前記複数の映像領域データのそれぞれに対するデータ低減量を設定し、
    　当該設定された前記複数の映像領域データごとのデータ低減量に基づいて低減された送信用の映像データを生成し、
    　当該送信用の映像データを前記通信装置を介して前記遠隔操作装置に送信することを特徴とする作業機械。     a working device; a photographing device that photographs images of the working device and its surroundings; and transmitting the video photographed by the photographing device as video data to a remote control device via a communication device; A working machine comprising: a control device that controls the operation of the working device according to an operation signal received via a communication device;
    The control device includes:
    receiving from the remote operating device an evaluation result of communication quality with the remote operating device via the communication device;
    determining the operating status of the working device based on the operating signal;
    Based on the evaluation result of the communication quality and the operation status of the work device, the video data is divided into a plurality of video area data corresponding to a plurality of video areas of the video, and the video data for each of the plurality of video area data is Set the data reduction amount,
    Generate reduced video data for transmission based on the set data reduction amount for each of the plurality of video area data,
    A work machine characterized in that the video data for transmission is transmitted to the remote control device via the communication device.
  2. 　請求項１に記載の作業機械において、
    　前記制御装置は、
    　前記通信品質が低いことを示す評価結果を受信し、かつ前記作業装置が動作していると判定した場合、
    　前記映像データを、前記映像のうちの前記作業装置の先端及びその周囲を含む主の映像領域に対応する主映像領域データと他の映像領域に対応する副映像領域データとに分割すると共に、前記主映像領域データより前記副映像領域データの低減量又は低減率を大きくするか、若しくは前記主映像領域データのデータ量より前記副映像領域データのデータ量方が小さくなるようにデータ低減を行うことを特徴とする作業機械。     The working machine according to claim 1,
    The control device includes:
    When receiving an evaluation result indicating that the communication quality is low and determining that the work device is operating,
    dividing the video data into main video area data corresponding to a main video area including the tip of the working device and its surroundings in the video, and sub-video area data corresponding to other video areas; Data reduction is performed such that the reduction amount or reduction rate of the sub-video area data is greater than that of the main video area data, or the data amount of the sub-video area data is smaller than the data amount of the main video area data. A working machine featuring:
  3. 　請求項２に記載の作業機械において、
    　前記制御装置は、前記作業装置を含む映像領域を基準として前記作業装置の移動方向側の映像領域がその反対方向側の映像領域より大きくなるように、前記主の映像領域の形状を設定することを特徴とする作業機械。     The working machine according to claim 2,
    The control device sets the shape of the main video area so that the video area on the side of the moving direction of the work device is larger than the video area on the opposite direction with respect to the video area including the work device. A working machine featuring:
  4. 　請求項１に記載の作業機械において、
    　走行体と、前記走行体に対して旋回可能に支持された旋回体と、を有し、
    　前記制御装置は、
    　前記操作信号に応じて前記走行体及び前記旋回体の動作を制御し、
    　前記操作信号に基づいて前記走行体及び前記旋回体の動作状況を判定し、
    　前記通信品質が低いことを示す評価結果を受信し、かつ前記旋回体が動作していると判定した場合、
    　前記映像データを、前記映像のうちの前記作業装置の移動先を含む主の映像領域に対応する主映像領域データと他の映像領域に対応する副映像領域データとに分割すると共に、前記主映像領域データより前記副映像領域データのデータ低減量又は低減率を大きくするか、若しくは前記主映像領域データのデータ量より前記副映像領域データのデータ量の方が小さくなるようにデータ低減を行うことを特徴とする作業機械。     The working machine according to claim 1,
    comprising a traveling body and a rotating body rotatably supported with respect to the traveling body,
    The control device includes:
    controlling operations of the traveling body and the rotating body according to the operation signal;
    determining the operating status of the traveling body and the rotating body based on the operation signal;
    When receiving an evaluation result indicating that the communication quality is low and determining that the revolving structure is operating,
    The video data is divided into main video area data corresponding to a main video area including the movement destination of the work equipment in the video and sub-video area data corresponding to other video areas, and the main video Data reduction is performed so that the data reduction amount or reduction rate of the sub-video area data is larger than that of the area data, or the data amount of the sub-video area data is smaller than the data amount of the main video area data. A working machine featuring:
  5. 　請求項１に記載の作業機械において、
    　前記制御装置は、
    　前記撮影装置で撮影された映像の処理若しくは移動体検知センサの検知によって前記作業機械の周囲に存在する移動体を検知し、
    　前記作業機械の周囲に前記移動体が存在することを検知した場合、
    　前記映像データを、前記映像のうちの前記移動体及びその周囲を含む主の映像領域に対応する主映像領域データと他の映像領域に対応する副映像領域データとに分割すると共に、前記主映像領域データより前記副映像領域データのデータ低減量又は低減率を大きくするか、若しくは前記主映像領域データのデータ量より前記副映像領域データのデータ量の方が小さくなるようにデータ低減を行うことを特徴とする作業機械。     The working machine according to claim 1,
    The control device includes:
    Detecting a moving object existing around the work machine by processing an image taken by the photographing device or detecting a moving object detection sensor,
    When it is detected that the moving object exists around the working machine,
    The video data is divided into main video area data corresponding to a main video area including the moving body and its surroundings in the video, and sub-video area data corresponding to other video areas, and Data reduction is performed so that the data reduction amount or reduction rate of the sub-video area data is larger than that of the area data, or the data amount of the sub-video area data is smaller than the data amount of the main video area data. A working machine featuring:
  6. 　請求項４に記載の作業機械において、
    　前記制御装置は、
    　前記作業機械の動作パターンを学習して記憶し、前記作業機械の動作パターンを用いて、現在の前記作業機械の動作又は前記作業機械の周囲における移動体の存在から、将来の前記作業装置の動作及び前記旋回体の動作のうちの少なくとも一方を予測し、
    　予測された前記作業装置の動作及び前記旋回体の動作のうちの少なくとも一方に応じて、前記映像データを前記映像の複数の映像領域に対応する複数の映像領域データに分割すると共に前記複数の映像領域データのそれぞれに対するデータ低減量を設定することを特徴とする作業機械。     The working machine according to claim 4,
    The control device includes:
    The operation pattern of the work machine is learned and memorized, and the operation pattern of the work machine is used to determine the future operation of the work equipment based on the current operation of the work machine or the presence of moving objects around the work machine. and predicting at least one of the movements of the rotating body,
    The video data is divided into a plurality of video region data corresponding to a plurality of video regions of the video according to at least one of the predicted motion of the working device and the predicted motion of the revolving body, and A work machine characterized by setting a data reduction amount for each area data.
  7. 　作業装置、前記作業装置及びその周囲の映像を撮影する撮影装置、並びに前記撮影装置で撮影された映像を映像データとして第１の通信装置を介して送信すると共に、前記第１の通信装置を介して受信した操作信号に応じて前記作業装置の動作を制御する第１の制御装置を備えた作業機械と、
    　種々の情報を表示する表示装置、操作者の操作に応じて前記操作信号を生成する操作装置、並びに前記作業機械から第２の通信装置を介して受信した前記映像データに対応する映像を前記表示装置に表示させると共に、前記操作装置で生成された前記操作信号を前記第２の通信装置を介して前記作業機械へ送信する第２の制御装置を備えた遠隔操作装置と、を備えた遠隔操作型作業機械システムにおいて、
    　前記遠隔操作装置の前記第２の制御装置は、前記作業機械との通信の品質を評価し、当該評価結果を前記作業機械に送信し、
    　前記作業機械の前記第１の制御装置は、
    　前記操作信号に基づいて、前記作業装置の動作状況を判定し、
    　前記遠隔操作装置から、前記作業機械と前記遠隔操作装置の間の通信品質が低下していることを示す評価結果を受信し、かつ前記作業装置が動作していると判定した場合、前記映像データを前記映像の複数の映像領域に対応する複数の映像領域データに分割すると共に、前記複数の映像領域データのそれぞれに対するデータ低減量を設定し、
    　当該設定された前記複数の映像領域データごとのデータ低減量に基づいて低減された送信用の映像データを生成し、
    　当該送信用の映像データを前記遠隔操作装置に送信することを特徴とする遠隔操作型作業機械システム。     A working device, a photographing device that photographs images of the working device and its surroundings, and transmitting the video photographed by the photographing device as video data via a first communication device, and a working machine comprising a first control device that controls the operation of the working device according to an operation signal received by the working machine;
    a display device that displays various information; an operating device that generates the operation signal in response to an operation by an operator; and a display device that displays a video corresponding to the video data received from the working machine via a second communication device. a remote control device including a second control device that displays the operation signal on the device and transmits the operation signal generated by the operation device to the work machine via the second communication device. In the mold working machine system,
    The second control device of the remote control device evaluates the quality of communication with the working machine, and transmits the evaluation result to the working machine,
    The first control device of the work machine includes:
    determining the operating status of the working device based on the operating signal;
    If an evaluation result indicating that the communication quality between the work machine and the remote control device has deteriorated is received from the remote control device, and it is determined that the work device is operating, the video data dividing into a plurality of video region data corresponding to a plurality of video regions of the video, and setting a data reduction amount for each of the plurality of video region data,
    Generate reduced video data for transmission based on the set data reduction amount for each of the plurality of video area data,
    A remote-controlled work machine system, characterized in that the video data for transmission is transmitted to the remote control device.
PCT/JP2023/010201 2022-03-31 2023-03-15 Work machine and remote-controlled work machine system WO2023189631A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2022058424A JP2023149698A (en) 2022-03-31 2022-03-31 Work machine and remote control type work machine system
JP2022-058424 2022-03-31

Publications (1)

Publication Number Publication Date
WO2023189631A1 true WO2023189631A1 (en) 2023-10-05

Family

ID=88201635

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2023/010201 WO2023189631A1 (en) 2022-03-31 2023-03-15 Work machine and remote-controlled work machine system

Country Status (2)

Country Link
JP (1) JP2023149698A (en)
WO (1) WO2023189631A1 (en)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2017043924A (en) * 2015-08-25 2017-03-02 日立建機株式会社 Remote control system for construction machinery
JP2018106676A (en) * 2016-12-22 2018-07-05 パナソニック インテレクチュアル プロパティ コーポレーション オブ アメリカＰａｎａｓｏｎｉｃ Ｉｎｔｅｌｌｅｃｔｕａｌ Ｐｒｏｐｅｒｔｙ Ｃｏｒｐｏｒａｔｉｏｎ ｏｆ Ａｍｅｒｉｃａ Information processing device, operated vehicle, information processing method, and program
JP2020200659A (en) * 2019-06-10 2020-12-17 コベルコ建機株式会社 Remote operation system and remote operation server
JP2021156030A (en) * 2020-03-27 2021-10-07 日立建機株式会社 Remote control system of work machine

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2017043924A (en) * 2015-08-25 2017-03-02 日立建機株式会社 Remote control system for construction machinery
JP2018106676A (en) * 2016-12-22 2018-07-05 パナソニック インテレクチュアル プロパティ コーポレーション オブ アメリカＰａｎａｓｏｎｉｃ Ｉｎｔｅｌｌｅｃｔｕａｌ Ｐｒｏｐｅｒｔｙ Ｃｏｒｐｏｒａｔｉｏｎ ｏｆ Ａｍｅｒｉｃａ Information processing device, operated vehicle, information processing method, and program
JP2020200659A (en) * 2019-06-10 2020-12-17 コベルコ建機株式会社 Remote operation system and remote operation server
JP2021156030A (en) * 2020-03-27 2021-10-07 日立建機株式会社 Remote control system of work machine

Also Published As

Publication number Publication date
JP2023149698A (en) 2023-10-13

Similar Documents

Publication Publication Date Title
US11873620B2 (en) Turning control apparatus for turning-type working machine
US20220010519A1 (en) Shovel and construction system
JP2006219894A (en) Camera controlling device in remote control of construction machinery
KR20200133721A (en) Shovel
WO2021054417A1 (en) Excavator, excavator management device, excavator management system, and excavator assist device
US10889958B2 (en) Display system for machine
JP5466536B2 (en) Work machine monitoring device
WO2023189631A1 (en) Work machine and remote-controlled work machine system
US11537121B2 (en) Work vehicle, control device, and method for controlling work vehicle
JP7063036B2 (en) Construction machinery
WO2021230093A1 (en) Image processing device and image processing method
JPH08333086A (en) Processor of photographed picture image of hung cargo
US20220205224A1 (en) Excavator and management apparatus for excavator
JP7376440B2 (en) Remote control device and remote control system
US20220275604A1 (en) Remote control system for work machine
JP3549332B2 (en) Automatic shooting camera system
JP6689772B2 (en) Excavator
JPH10329070A (en) Method for remote wireless control and device therefor
JP3611394B2 (en) Camera direction control device for work machines
JPH09193078A (en) Camera direction control device of remote control machine
WO2021020292A1 (en) Display system, remote operation system, and display method
WO2023112217A1 (en) Video transmission system, video transmission device, and video transmission method
JPH09247515A (en) Camera visual field angle controller for remote-controlled machine
WO2023228897A1 (en) Remote control system, remote-operated work machine system, and work information display control method
WO2021141077A1 (en) Inspection system for construction machine

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23779649

Country of ref document: EP

Kind code of ref document: A1