WO2023100533A1

WO2023100533A1 - Image display system, remote operation assistance system, and image display method

Info

Publication number: WO2023100533A1
Application number: PCT/JP2022/039550
Authority: WO
Inventors: 卓伊藤; 洋一郎山▲崎▼; 誠司佐伯; 佑介上村; 友鷹三谷
Original assignee: コベルコ建機株式会社
Priority date: 2021-12-03
Filing date: 2022-10-24
Publication date: 2023-06-08
Also published as: JP2023083245A

Abstract

Provided are a system and the like that can improve accuracy of recognition, by an operator, for the positional relationship between a work mechanism constituting a work machine and an object located in the vicinity of the work machine. According to the present invention, an operator can grasp the position of a second index point p2 on the surface of an object Obj and the position of a work mechanism 440 (attachment 445) constituting a work machine 40, by means of a work environment image outputted to a remote image output device 221 constituting a remote output interface 220 and an index image M superimposed on the work environment image. The second index point p2 is a result obtained by projecting a first index point p1 on the surface of the object Obj.

Description

Image display system, remote control support system and image display method

The present invention relates to a technique for supporting remote control by an operator of a work machine such as a hydraulic excavator.

Using information on the position of the work implement obtained using the attitude of the work machine and information on the position of the ground, which is the work target, obtained from the information on the distance to the work target of the work machine obtained by the distance detection device. proposed a technique for generating an image of a portion corresponding to the work tool along the surface of the work object facing the work tool, synthesizing it with the image of the work object captured by an imaging device, and displaying the image on a display device. (See Patent Document 1, for example). This suppresses a decrease in work efficiency when working with a work machine having a work implement having a work implement.

Japanese Patent No. 6777375

However, when the work device and the work target are aligned, according to the prior art, the position of the work device is represented by an image formed along the terrain. Therefore, since unnecessary positional information other than the predetermined part of the working device is also displayed at the same time, it is difficult to grasp the position of the working device with respect to the work target, which may make it difficult to perform the work efficiently. have a nature.

Therefore, an object of the present invention is to provide a system and the like that can improve the accuracy of recognition by an operator of the positional relationship between a working mechanism that constitutes a working machine and a target object that exists around the working machine.

The image display system of the present invention is
In a working environment image representing the situation of a working mechanism that constitutes a working machine and a target object that exists around the working machine, a second index point obtained as a result of projecting a first index point in the working mechanism onto the surface of the target object An image display system for superimposing index images indicating index points and outputting them to an output interface of a remote control device for remotely controlling the working machine.

According to the image display system having this configuration, the second index points on the surface of the working mechanism and the target object that constitute the working machine are displayed through the working environment image output to the output interface of the remote control device and the index image superimposed thereon. can be made to grasp the position of the operator. The second index point is the result of projecting the first index point onto the surface of the target object and does not follow the surface shape of the target object. Therefore, it is possible to avoid providing the operator with unnecessary positional information of the working mechanism other than the first index point, thereby improving the operator's recognition accuracy of the positional relationship between the working mechanism and the target object.

FIG. 2 is an explanatory diagram regarding the configuration of an image display composite system and an image display system; Explanatory drawing about the structure of a remote control. Explanatory drawing about the structure of a working machine. Explanatory drawing about the function of an image display system. FIG. 4 is an explanatory diagram of one display form of a work environment image and index images; FIG. 4 is an explanatory diagram regarding the relationship between the arm top position and the display mode of the index image; FIG. 4 is an explanatory diagram regarding the relationship between the arm top position and the display mode of the index image; FIG. 4 is an explanatory diagram regarding the relationship between the arm top position and the display mode of the index image; FIG. 5 is an explanatory diagram regarding the relationship between the displacement mode of the working mechanism and the directivity of the index image; FIG. 5 is an explanatory diagram regarding the relationship between the displacement mode of the working mechanism and the directivity of the index image; FIG. 5 is an explanatory diagram regarding the relationship between the displacement mode of the working mechanism and the directivity of the index image; FIG. 5 is an explanatory diagram relating to the relationship between the posture mode of the working mechanism and the directivity of the index image; FIG. 5 is an explanatory diagram relating to the relationship between the posture mode of the working mechanism and the directivity of the index image; FIG. 5 is an explanatory diagram relating to the relationship between the posture mode of the working mechanism and the directivity of the index image; FIG. 4 is an explanatory diagram of a display form of a three-dimensional index image; FIG. 4 is an explanatory diagram of a display form of a three-dimensional index image; FIG. 4 is an explanatory diagram of a display form of a three-dimensional index image; FIG. 5 is an explanatory diagram regarding the relationship between the displacement mode of the working mechanism and the position of the first index point; FIG. 5 is an explanatory diagram regarding the relationship between the displacement mode of the working mechanism and the position of the first index point; FIG. 5 is an explanatory diagram regarding the relationship between the displacement mode of the working mechanism and the position of the first index point; FIG. 11 is an explanatory diagram of another display form of the working environment image and the index image; FIG. 4 is an explanatory diagram regarding the relationship between the operation mode of the working mechanism and the display mode of the index image; FIG. 4 is an explanatory diagram regarding the relationship between the operation mode of the working mechanism and the display mode of the index image; FIG. 5 is an explanatory diagram regarding the relationship between the displacement mode of the working mechanism and the position of the first index point; FIG. 5 is an explanatory diagram regarding the relationship between the displacement mode of the working mechanism and the position of the first index point;

(Configuration of image display complex system)
The composite image display system shown in FIG. 1 includes an image display system 10, a remote control device 20, and/or a work machine 40 to be remotely controlled by the remote control device 20. As shown in FIG. The image display system 10, the remote controller 20 and the work machine 40 are configured to be able to communicate with each other through a network. The mutual communication network of image display system 10 and remote control device 20 and the mutual communication network of image display system 10 and work machine 40 may be the same or different.

(Configuration of image display system)
In this embodiment, the image display system 10 is configured by a computer that exists separately from the remote control device 20 and the work machine 40, and includes a database 102, a communication function element 121, and an image processing function element 122. I have. The database 102 stores and holds captured image data and the like. The database 102 may be composed of a database server capable of mutual communication with the image display system 10 . Each functional element is composed of an arithmetic processing unit (for example, a single-core processor and/or a multi-core processor or a processor core that constitutes this), reads necessary data and software from a storage device such as a memory, and targets the data Calculation processing, which will be described later, is executed according to the software.

(Configuration of remote control device)
The remote control device 20 includes a remote control device 200 , a remote input interface 210 , a remote output interface 220 and a remote wireless communication device 224 . The remote control device 200 is configured by an arithmetic processing device (for example, a single-core processor and/or a multi-core processor or a processor core constituting this), reads necessary data and software from a storage device such as a memory, and processes the data. , the arithmetic processing according to the software is executed.

The remote input interface 210 includes a remote control mechanism 211. Remote output interface 220 includes a remote image output device 221 .

The remote control mechanism 211 includes a traveling operating device, a turning operating device, a boom operating device, an arm operating device, and a bucket operating device. Each operating device has an operating lever that receives a rotating operation. An operating lever (running lever) of the operating device for running is operated to move the lower running body 410 of the work machine 40 . The travel lever may also serve as a travel pedal. For example, a traction pedal may be provided that is fixed to the base or lower end of the traction lever. An operation lever (swing lever) of the swing operation device is operated to move a hydraulic swing motor that constitutes the swing mechanism 430 of the work machine 40 . An operating lever (boom lever) of the boom operating device is operated to move the boom cylinder 442 of the work machine 40 . An operating lever (arm lever) of the arm operating device is operated to move the arm cylinder 444 of the work machine 40 . An operating lever (bucket lever) of the bucket operating device is operated to move the bucket cylinder 446 of the work machine 40 .

For example, as shown in FIG. 2, each control lever that constitutes the remote control mechanism 211 is arranged around the seat St on which the operator sits. The seat St is in the form of a high-back chair with armrests, a low-back chair without a headrest, or a chair without a backrest. may be

A pair of left and right travel levers 2110 corresponding to the left and right crawlers are arranged side by side in front of the seat St. One operating lever may serve as a plurality of operating levers. For example, the left operation lever 2111 provided in front of the left frame of the seat St shown in FIG. 2 functions as an arm lever when operated in the longitudinal direction, and when operated in the lateral direction. function as a pivot lever. Similarly, the right operation lever 2112 provided in front of the right frame of the seat St shown in FIG. It may function as a bucket lever in some cases. The lever pattern may be arbitrarily changed by an operator's operation instruction.

For example, as shown in FIG. 2, the remote image output device 221 includes a central remote image output device 2210 having substantially rectangular screens arranged in front, diagonally forward left, and diagonally forward right of the sheet St. It consists of a left remote image output device 2211 and a right remote image output device 2212 . The shapes and sizes of the respective screens (image display areas) of the central remote image output device 2210, the left remote image output device 2211 and the right remote image output device 2212 may be the same or different.

As shown in FIG. 2, the left remote image is tilted such that the screen of the central remote image output device 2210 and the screen of the left remote image output device 2211 form an inclination angle θ1 (for example, 120°≦θ1≦150°). The right edge of output device 2211 is adjacent to the left edge of central remote image output device 2210 . As shown in FIG. 2, the right remote image is tilted such that the screen of the central remote image output device 2210 and the screen of the right remote image output device 2212 form an inclination angle θ2 (eg, 120°≦θ2≦150°). The left edge of output device 2212 is adjacent to the right edge of central remote image output device 2210 . The tilt angles θ1 and θ2 may be the same or different.

The respective screens of the central remote image output device 2210, the left remote image output device 2211, and the right remote image output device 2212 may be parallel to the vertical direction or may be inclined with respect to the vertical direction. At least one of the central remote image output device 2210, the left remote image output device 2211 and the right remote image output device 2212 may be composed of a plurality of divided image output devices. For example, the central remote image output device 2210 may comprise a pair of vertically adjacent image output devices having substantially rectangular screens.

The remote image output device 221 may be composed of a single image output device curved or bent to surround the sheet St. A single image output device may be constituted by, for example, the central remote image output device 2210 . The remote image output device 221 may consist of two image output devices (eg, a central remote image output device 2210 and a left remote image output device 2211 or a right remote image output device 2212).

(Configuration of working machine)
The working machine 40 includes a real machine control device 400 , a real machine input interface 41 , a real machine output interface 42 , a real machine wireless communication device 422 , and a working mechanism 440 . The actual device control device 400 is composed of an arithmetic processing device (single-core processor or multi-core processor or a processor core that constitutes it), reads necessary data and software from a storage device such as a memory, and processes the data into the software. Execute the arithmetic processing accordingly.

The work machine 40 is, for example, a crawler excavator (construction machine), and as shown in FIG. and an upper revolving body 420 . A cab 424 (driver's cab) is provided on the front left side of the upper swing body 420 . A work mechanism 440 is provided in the front central portion of the upper swing body 420 .

The real machine input interface 41 includes a real machine operating mechanism 411 , a real machine imaging device 412 , a real machine distance measuring device 414 , and a real machine sensor group 416 . The actual machine operating mechanism 411 includes a plurality of operating levers arranged around a seat arranged inside the cab 424 in the same manner as the remote operating mechanism 211 . The cab 424 is provided with a drive mechanism or a robot that receives a signal corresponding to the operation mode of the remote control lever and moves the actual machine control lever based on the received signal. The actual machine imaging device 412 is installed inside the cab 424, for example, and images the environment including at least part of the working mechanism 440 (for example, the attachment 445) through the front window. Some or all of the front window and side windows may be omitted. The real machine range finder 414 is a device for measuring the real space distance to a target object existing around the work machine 40, and thus the real space position, and is composed of, for example, a LiDAR and a TOF sensor. The actual machine sensor group 416 includes a turning angle sensor for measuring the turning angle of the upper turning body 420 with respect to the lower traveling body 410, an attitude angle sensor for measuring the attitude angle representing the attitude of the working mechanism 440, and the like. It consists of various sensors for measuring 40 operating conditions.

The real machine output interface 42 includes a real machine wireless communication device 422 .　

As shown in FIG. 3, a working mechanism 440 as a working mechanism includes a boom 441 attached to the upper rotating body 420 so as to be able to rise and fall, and an arm 443 rotatably connected to the tip of the boom 441. and an attachment 445 (for example, a bucket) rotatably connected to the tip of the arm 443 . The working mechanism 440 is equipped with a boom cylinder 442, an arm cylinder 444, and a bucket cylinder 446, which are configured by telescopic hydraulic cylinders.

The boom cylinder 442 is interposed between the boom 441 and the upper slewing body 420 so as to expand and contract when supplied with hydraulic oil and rotate the boom 441 in the hoisting direction. The arm cylinder 444 is interposed between the arm 443 and the boom 441 so as to expand and contract when supplied with hydraulic oil to rotate the arm 443 about the horizontal axis with respect to the boom 441 . The bucket cylinder 446 is interposed between the attachment 445 and the arm 443 so as to expand and contract when supplied with hydraulic oil to rotate the attachment 445 with respect to the arm 443 about the horizontal axis.

(function)
FIG. 4 is a flow chart for explaining the image display system having the above configuration and the function of the image display system. In the flowchart, the block "C●" is used for simplification of the description, means transmission and/or reception of data, and processing in the branch direction is executed on the condition of transmission and/or reception of the data. It means a conditional branch. The flowchart is repeated for each control cycle, and after reaching "END", the process returns to "START" and the subsequent processes are executed.

In the remote operation device 20, it is determined whether or not the operator has performed an environment confirmation request operation (second specified operation) through the remote input interface 210 (FIG. 4/STEP 210). An "environment confirmation request" is, for example, an operation such as a tap on the remote input interface 210 for the operator to instruct the work machine 40 intended to be remotely operated to perform an environment confirmation request operation. If the determination result is negative (FIG. 4/STEP 210 . . . NO), return to START. On the other hand, if the determination result is affirmative (FIG. 4/STEP 210 . . . YES), an environment confirmation request is transmitted to the image display system 10 through the remote wireless communication device 224 (FIG. 4/STEP 211).

When the image display system 10 receives an environment confirmation request, the communication function element 121 transmits the environment confirmation request to the relevant work machine 40 (FIG. 4/C10). The environment confirmation request may be transmitted to work machine 40 without going through image display system 10 .

When the work machine 40 receives an environment confirmation request through the actual machine wireless communication device 422 (FIG. 4/C40), the actual machine imaging device 412 detects the work object Obj (for example, the ground, earth and sand, materials and/or buildings) are acquired, a three-dimensional image of the work object Obj is acquired by the actual machine distance measuring device 414, and three-dimensional image data representing the three-dimensional image is transmitted to the actual machine wirelessly. It is transmitted to the image display system 10 through the communication device 422 (FIG. 4/STEP 410).

A three-dimensional image is an image having the direction and distance to the work object Obj or the real space position of the work object Obj acquired through the actual rangefinder 414 . A “real space position” is defined by coordinate values (for example, latitude, longitude and altitude) in a real space coordinate system or coordinate values in a real machine coordinate system (a coordinate system whose position or orientation is fixed with respect to the work machine 40). be. When the work object Obj is reflected in the captured image, the real space position of each point forming the point group on the surface of the work object Obj corresponding to each pixel of the three-dimensional image is the pixel value of each pixel. included as

The three-dimensional image data consists of data of a captured image obtained through the actual imaging device 412 or a model image equivalent thereto, distance or real space position data obtained through the actual ranging device 414, and may be acquired and transmitted as a combination of

A captured image including at least the work object Obj is not captured by the actual imaging device 412, but is captured by an imaging device installed around the working machine 40, an imaging device mounted on an unmanned airplane, and/or carried by a field worker. It may be acquired through an imaging device of the equipment that is being used. The distance or real space position, which is the pixel value of the three-dimensional image, may be obtained through a rangefinder installed around work machine 40 and/or a rangefinder mounted on an unmanned aerial vehicle.

Instead of the combination of the actual imaging device 412 and the actual distance measuring device 414, a stereo camera (a pair of left and right actual imaging devices 412) mounted on the work machine 40 is used to acquire a captured image and a three-dimensional image of the work object Obj. may

In the image display system 10, when three-dimensional image data is received by the communication function element 121 (FIG. 4/C11), the image processing function element 122 outputs work environment image data corresponding to the three-dimensional image data to the remote control device 20. (FIG. 4/STEP 110). The work environment image data includes the captured image data itself (which does not include the real space position and distance information as pixel values) that is the basis of the 3D image data, as well as the simulated image data generated based on the captured image data. This is image data representing a working environment image.

When the remote operation device 20 receives the working environment image data through the remote wireless communication device 224 (FIG. 4/C21), the remote control device 200 outputs the working environment image corresponding to the working environment image data to the remote image output device 221. (FIG. 4/STEP 212).

This allows, for example, as shown in FIG. A work environment image in which earth and sand, which are objects Obj, are reflected is output to the remote image output device 221 .

In working machine 40, real machine sensor group 416 acquires the real space position of first index point _p1 of working mechanism 440, and data representing the real space position of first index point _p1 is transmitted to the real machine wireless communication device. 422 to the image display system 10 (FIG. 4/STEP 412).

The process of transmitting three-dimensional image data (see STEP 410 in FIG. 4) and the process of transmitting data representing the real space position of the first index point p ₁ (see STEP 412 in FIG. 4) are simultaneously executed as a process of transmitting a set of data. may be

Specifically, a point corresponding to the tip (arm top) of the arm 443 is defined as the first index point _p1 . The real space position of the first index point p ₁ defined in the working mechanism 440 is the output signal of the attitude angle sensor that constitutes the real machine sensor group 416 mounted on the working machine 40, and the output signal of each working mechanism 440. Calculated forward kinematics based on component sizes. The attitude angle sensor detects at least part of the hoisting angle of the boom 441 with respect to the upper swing body 420, the rotation angle of the arm 443 at the joint with the boom 441, and the rotation angle of the attachment 445 at the joint with the arm 443. is configured to output a signal corresponding to An arbitrary point such as the boom 441, the arm 443 and the attachment 445 that constitute the working mechanism 440 may be defined as the first index point _p1 .

When the first index point p ₁ of the work mechanism 440 is reflected in the three-dimensional image, the real space position of the first index point p ₁ may be recognized by the actual machine control device 400 based on the three-dimensional image. . Specifically, image analysis processing (grayscaling processing, edge extraction processing, and/or pattern matching processing, etc.) of the three-dimensional image causes one or more pixels corresponding to the first index point p ₁ in the working mechanism 440 to be The average value of the pixel values is recognized as the real space position of the first index point _p1 . Based on one of the real space position of each point of working mechanism 440 recognized using the attitude angle sensor and the real space position that is the pixel value of the three-dimensional image, the other may be corrected.

When the image display system 10 receives the data representing the real space position of the first index point p ₁ (FIG. 4/C12), the real space position of the first index point p ₁ and the three-dimensional image include The real space position of the second index point p ₂ is recognized by the image processing functional element 122 based on the real space position or three-dimensional shape of each point forming the point group on the surface of the target object Obj (FIG. 4/STEP 112 ). The second index point _p2 is a point resulting from projection of the first index point _p1 onto the surface of the target object Obj. The projection direction of the first index point p ₁ with respect to the surface of the target object Obj is, for example, the vertical direction. In this case, among the points on the surface of the target object Obj, the real space position of one point whose horizontal position (x (longitude), y (longitude)) is the same as or closest to the first index point p ₁ , or The center of gravity of a plurality of points horizontally close to the first index point p ₁ is recognized as the second index point p ₂ .

When the work machine 40 or the upper revolving body 420 is tilted with respect to the vertical axis of the real space, the projection direction of the first index point _p1 with respect to the surface of the target object Obj is similarly with respect to the vertical axis of the real space. It may be defined in an oblique direction. The tilt angle of the work machine 40 with respect to the vertical axis is measured by a body tilt angle sensor (for example, a gyro sensor) that constitutes the actual machine sensor group 416 .

When the first index point p ₁ of the working mechanism 440 is reflected in the three-dimensional image, the recognition of the real space position of the first index point p ₁ by the real machine control device 400 and the transmission of data (FIG. 4/STEP 412) May be omitted. In this case, the image processing functional element 122 performs image analysis processing (such as grayscaling, edge extraction, and/or pattern matching) on the three-dimensional image to define the first index point p ₁ in the working mechanism 440. One or more pixels corresponding to the point are recognized, and the pixel value of the pixels or the average value thereof is recognized as the real space position of the first index point _p1 .

Further, in the image display system 10, the index image data in which the index image M indicating the second index point _p2 is superimposed on the working environment image is transmitted to the remote control device 20 by the image processing functional element 122 (Fig. 4/STEP 114). The command includes the real space position of the second index point _p2 and/or the pixel position (u, v) corresponding to the second index point _p2 in the three-dimensional image or work environment image.

When the remote control device 20 receives the index image data through the remote wireless communication device 224 (FIG. 4/C22), the remote control device 200 superimposes the index image M corresponding to the command on the working environment image. is output to the remote image output device 221 (FIG. 4/STEP 214).

The process of transmitting work environment image data (see FIG. 4/STEP 110) and the process of transmitting index image data (see FIG. 4/STEP 114) from the image display system 10 to the remote control device 20 are simultaneously executed as a batch of data transmission process. may be In this case, in the remote control device 20, output processing of the work environment image (see FIG. 4/STEP 212) and output processing of the index image superimposed on the work environment image (see FIG. 4/STEP 214) are performed as a single image output. They may be executed simultaneously as processes.

As a result, for example, as shown in FIG. 5, a second index is obtained as a result of projecting the first index point p ₁ onto the surface of the target object Obj (for example, earth and sand around the working machine 40). The index image M indicating the point _p2 is output to the remote image output device 221 in a form superimposed on the working environment image. Here, the index image M is an image of a triangular or arrow-shaped figure pointing toward the second index point _p2 in the vertical direction in real space. As shown in FIG. 5, the first index point _p1 and the second index point _p2 may be superimposed on the captured image, and the first index point _p1 and/or the second index point _p2 The superimposed display may be omitted.

In the remote control device 20, the remote control device 200 recognizes the operation mode of the remote control mechanism 211, and transmits a remote control command corresponding to the operation mode to the image display system 10 through the remote wireless communication device 224. (FIG. 4/STEP 220).

In the image display system 10, when the image processing functional element 122 receives the remote operation instruction, the communication functional element 121 transmits the remote operation instruction to the working machine 40 (FIG. 4/C14). The remote control command may be transmitted to work machine 40 without going through image display system 10 .

In the work machine 40, when the actual machine control device 400 receives an operation command through the actual machine wireless communication device 422 (FIG. 4/C44), the operation of the working mechanism 440 and the like is controlled (FIG. 4/STEP 420). For example, the attachment 445 scoops up earth and sand, which is the target object Obj in front of the working machine 40 , rotates the upper revolving body 420 , and then removes the earth and sand from the attachment 445 .

(Effect)
According to the image display system that constitutes the image display system with this configuration, the work machine 40 can be viewed through the work environment image output to the remote image output device 221 that constitutes the remote output interface 220 and the index image M superimposed thereon. The operator can grasp the working mechanism 440 (attachment 445) and the position of the second index point _p2 on the surface of the target object Obj (see FIG. 5). The second index point _p2 is the _result of projecting the first index point _p1 onto the surface of the target object Obj. This prevents the operator from being provided with the information, thereby improving the recognition accuracy by the operator of the positional relationship between the working mechanism 440 and the target object Obj.

6A to 6C each show the positional relationship between the arm top and the attachment 445 when excavating the ground. Working mechanism 440 can exert the strongest force under the arm top, which is the connection point for attachment 445 . Taking this into account, the operator usually contacts the tip of the attachment 445 with the ground below or farther from the arm top, and then moves the attachment 445 so that the tip is below the arm top. Ideally, operate so that the tip is closer to you than the arm top. In other words, when considering a series of excavation operations by the attachment 445 , the arm top is often more suitable than the tip of the attachment 445 as the position index of the working mechanism 440 .

In addition, the attachment 445 can be replaced with a breaker, grapple, lifting magnet, etc. in addition to the bucket described above, but since the position of the "arm top" does not change even if the attachment 445 is replaced, the same image display can be applied. There are advantages.

Also, the index image M is an image that has directivity with respect to the second index point _p2 on the surface of the target object Obj, or that indicates the position by a substantially triangular vertex or an arrow (see FIG. 5). Therefore, while avoiding the operator's mistake regarding the three-dimensional shape of the surface of the target object Obj, the operator can determine the positional relationship between the working mechanism 440, particularly the attachment 445 in which the first index point _p1 is defined, and the target object Obj. The ease of recognition is improved by

(Another embodiment of the present invention)
In the above-described embodiment, the image display system 10 and the communication function element 121 and the image processing function element 122 that constitute it are configured by a computer that exists separately from the remote control device 20 and the work machine 40, but other embodiments , an image display system is mounted on the remote control device 20 and/or the work machine 40, and the communication function element 121 and/or the image processing function element 122 are configured by the remote control device 200 and/or the actual machine control device 400. good. In that case, the communication function in the image display system 10 can be omitted.

Although the first index point p ₁ was defined as the arm top in the above embodiment, it may be defined at the tip of the attachment 445 . In that case, the index image M showing the second index point p ₂ as a result of projecting the first index point p ₁ onto the surface of the target object Obj allows the operator to determine the position of the attachment 445 with respect to the contact of the target object Obj. It is possible to improve the recognition accuracy of the relationship.

In response to the index image output command, the first index point p ₁ moves toward the target object Obj in a direction corresponding to the displacement mode of the attachment (for example, the attachment 445) on which the first index point p ₁ is arranged in the working mechanism 440. An index image M showing the second index point p ₂ as a result of being projected onto the surface may be superimposed on the working environment image and output to the remote image output device 221 . The displacement mode of the working mechanism 440 or the attachment is recognized based on the attitude angle sensor and/or the turning angle sensor that constitute the actual machine sensor group 416 . The displacement mode of the working mechanism 440 or the attachment may be recognized based on the operation mode of the control lever that constitutes the remote control mechanism 211 .

For example, as shown in FIG. 7A, when the attachment 445 is displaced vertically downward, the first index point p ₁ is projected vertically downward onto the surface of the target object Obj, resulting in the second An index image M indicating the index point p ₂ is superimposed on the work environment image and output to the remote image output device 221 . As shown in FIG. 7B, when the attachment 445 is displaced forward and downward as viewed from the work machine 40, the first index point _p1 is forward and downward of the work machine 40 with respect to the surface of the target object Obj. An index image M indicating the second index point p ₂ as a result of the projection is superimposed on the working environment image and output to the remote image output device 221 . As shown in FIG. 7C, when the attachment 445 is displaced rearward and downward as viewed from the work machine 40, the first index point p ₁ is rearward and downward of the work machine 40 with respect to the surface of the target object Obj. An index image M indicating the second index point p ₂ as a result of the projection is superimposed on the working environment image and output to the remote image output device 221 .

According to the image display system having this configuration, the displacement mode of the working mechanism 440 constituting the working machine 40 is displayed through the working environment image output to the remote image output device 221 of the remote controller 20 and the index image M superimposed thereon. To improve the operator's recognition accuracy for the positional relationship between the first index point _p1 of the working mechanism 440 and the second index point _p2 on the surface of the target object Obj in the working environment image in the direction corresponding to can be done.

A second index point p ₂ as a result of projecting the first index point p ₁ onto the surface of the target object Obj in a direction corresponding to the posture mode of the working mechanism 440 or the attachment 445 according to the index image output command. may be superimposed on the working environment image and output to the remote image output device 221 .

For example, as shown in FIG. 8A, when the chisel, which is the hitting portion of the breaker 445, which is the attachment 445 having directivity to the target, is oriented vertically downward. , the index image M showing the second index point _p2 as a result of projecting the first index point _p1 vertically downward onto the surface of the target object Obj is superimposed on the work environment image, and is output to the remote image output device 221. output. As shown in FIG. 8B, when the chisel of breaker 445 is oriented forward and downward as viewed from work machine 40, first index point p ₁ is in front of work machine 40 with respect to the surface of target object Obj. An index image M indicating the second index point p ₂ as a result of downward projection is superimposed on the work environment image and output to the remote image output device 221 . As shown in FIG. 8C, when the chisel of breaker 445 is oriented rearward and downward as viewed from work machine 40, first index point _p1 is positioned rearward of work machine 40 with respect to the surface of target object Obj. An index image M indicating the second index point p ₂ as a result of downward projection is superimposed on the work environment image and output to the remote image output device 221 .

According to the image display system having this configuration, the work mechanism 440 or the attachment 445 constituting the work machine 40 is displayed through the work environment image output to the remote image output device 221 of the remote operation device 20 and the index image M superimposed thereon. Since the operator can easily recognize the position and direction acting on the object according to the posture of the operator, the recognition accuracy of the operator can be improved, and the work efficiency can be improved.

The first index point is defined in the attachment 445, and while the transmission of a specific remote control command is started and continues (the specific operation continues), the position information of the first index point is The position of the first index point _p1 immediately before the operation command is transmitted may be maintained without being updated.

For example, an operation switch that operates a breaker 445 that is an attachment 445 is provided on the lever of the remote control mechanism, and when the breaker 445 is operated by the remote control mechanism 211, the working machine 40 is controlled by the remote control device 20. A remote control command to operate the breaker 445 is sent to the image display system 10, and an update stop signal is sent to the image display system 10 so as not to update the position information of the first index point _p1 . Also, the update stop signal is continuously transmitted while the operation switch is being pressed. As a result, the positional information of the first index point _p1 is maintained while the positional information immediately before the operation switch is operated continues. When the operation switch is no longer pressed, the update stop signal is no longer sent to the image display system 10 . As a result, updating of the position information of the first index point _p1 is resumed.

According to the image display system with this configuration, the position information of the first index point is not updated while the attachment 445 is being operated. Therefore, the work environment image output to the remote image output device 221 of the remote control device 20 and the index image M superimposed thereon are vibrated at the first index point by the vibration of the attachment 445 caused by the operation of the attachment 445. vibrate by doing As a result, it is possible to prevent a phenomenon in which the operator, who is performing remote control while gazing at the index image M, becomes sick.

As described above, the second index point _p2 is defined as a result of projecting the first index point _p1 onto the surface of the target object Obj in a direction corresponding to the displacement mode or posture mode of the working mechanism 440 or attachment. In this case, a stereoscopic index image M may be output to the remote image output device 221 of the remote control device 20 in the work environment image.

For example, when the second index point p ₂ is defined as a result of projecting the first index point p ₁ vertically downward onto the surface of the target object Obj (see FIGS. 7A and 8A), FIG. As shown, a substantially conical index image M with a central axis parallel to the vertical direction of the real space and with the apex directed downward in the real space is superimposed on the work environment image and output to the remote image output device 221. be done. When the second index point p ₂ is defined as a result of projecting the first index point p ₁ onto the surface of the target object Obj to the front and lower side of the work machine 40 (see FIGS. 7B and 8B), the figure As shown in 9B, a substantially conical index image M whose apex is directed forward and downward in real space is superimposed on the work environment image and output to the remote image output device 221 . When the second index point p ₂ is defined as a result of projecting the first index point p ₁ on the surface of the target object Obj to the rear and downward direction of the work machine 40 (see FIGS. 7C and 8C), FIG. 9C 2, a substantially conical index image M whose apex is directed backward and downward in the real space is superimposed on the work environment image and output to the remote image output device 221 .

According to the image display system having this configuration, the index image M output to the remote image output device 221 of the remote control device 20 is a three-dimensional image. The operator can easily grasp the positional relationship between the first index point _p1 and the second index point _p2 on the surface of the target object Obj.

An index image M showing a second index point _p2 as a result of projecting the first index point _p1 whose position differs according to the displacement mode of the working mechanism 440 onto the surface of the target object Obj is superimposed on the work environment image. and output to the remote image output device 221 of the remote operation device 20 .

For example, when the upper swing structure 420 is not swinging relative to the lower carrier structure 410, the tip center point of the bucket 445 is defined as the first index point _p1 , as shown in FIG. 10A. On the other hand, according to the operation mode of the remote control mechanism 211 and/or the actual machine operating mechanism 411, or based on the output signal of the actual machine sensor group 416, the upper revolving body 420 is turned against the lower traveling body 410 as viewed from above. If a clockwise turn is estimated or predicted or measured (see leftward white arrow), the point to the left of the tip of bucket 445 is defined as first index point p ₁ , as shown in FIG. 10B. be. Further, according to the operation mode of the remote control mechanism 211 and/or the actual machine operating mechanism 411, or based on the output signal of the actual machine sensor group 416, the upper rotating body 420 rotates clockwise with respect to the lower traveling body 410 when viewed from above. 10C, the point to the right of the tip of bucket 445 is defined as first index point p ₁ , as shown in FIG. 10C. In each case, an index image M showing a second index point _p2 as a result of projecting the first index point _p1 onto the surface of the target object Obj is superimposed on the work environment image to form a remote image. It is output to the output device 221 .

When operating the working machine 40, the part to be observed in the working mechanism 440 changes according to the changing mode of the position and/or posture of the working mechanism 440. For example, if the operation is for rotating the upper rotating body 420 with respect to the lower traveling body 410, the part is the leading edge in the rotating direction, and if the operation is for moving the working mechanism 440 away from the center of the machine, the working mechanism 440 It is the leading edge in the direction away from the machine center.

According to the image display system having this configuration, the position and position of the working mechanism 440 constituting the working machine 40 can be displayed through the working environment image output to the remote image output device 221 of the remote control device 20 and the index image M superimposed thereon. / Alternatively, it becomes easier to grasp the positional relationship between the part to be watched and the target object Obj according to the posture change mode, and the recognition accuracy of the operator can be improved.

A plurality of index images M each showing a plurality of second index points _p2 as a result of projecting the plurality of first index points _p1 onto the surface of the target object Obj are superimposed on the work environment image. It may be output to the remote image output device 221 of the remote operation device 20 .

For example, as shown in FIG. 11, the left and right endpoints of the tip of attachment 445 are defined as first index points p ₁₁ and p ₁₂ , respectively, and the first index points p ₁₁ and p A plurality of index images M ₁ and M ₂ representing second index points p ₂₁ and p ₂₂ , respectively, as a result of projecting each of ₁₂ vertically downward onto the surface of the target object Obj are superimposed on the work environment image. may be output to the remote image output device 221 of the remote operation device 20. The number of each of the first index points p ₁ and the corresponding second index points p ₂ may be 3 or more.

According to the image _display system with this configuration, the plurality _of first index points p The operator can grasp the position of each of the plurality of second index points _p21 and _p22 as a result of projecting each of ₁₁ and _p12 onto the surface of the target object Obj. As a result, compared to the case where only a single index image is output to the remote image output device 221, the operator's recognition accuracy of the positional relationship between the working mechanism 440 and the target object Obj is improved.

A plurality of index images oriented with respect to each of a plurality of second index points _p2 corresponding to _each of a plurality of first index points p1 whose relative positions change as the posture of the working mechanism 440 changes. M may be superimposed on the work environment image and output to the remote image output device 221 of the remote operation device 20 .

For example, as shown in FIG. 12A, when a pair of

members

4451 and 4452 that make up attachment 445 (eg, a grapple or crusher) are closed, the ends of the

respective members

4451 and 4452 of the pair are closed. A single index image M showing a second index point _p2 as a result of projecting the first index point _p1 onto the surface of the target object Obj is defined as the first designated point _p1 . The image is superimposed on the environment image and output to the remote image output device 221 of the remote operation device 20 . On the other hand, as shown in FIG. 12B, when the pair of

constituent members

4451 and 4452 that constitute the attachment 445 are opened, the distal ends of the pair of

constituent members

4451 and 4452 are positioned at the first designated point. a plurality of index images M ₁ defined as p ₁₁ and p ₁₂ and showing second index points p ₂₁ and p ₂₂ as a result of projecting the first index points p ₁₁ and p ₁₂ onto the surface of the target object Obj; M ₂ is superimposed on the work environment image and output to the remote image output device 221 of the remote controller 20 .

According to the image display system having this configuration, the working mechanism 440 (for example, the pair of

constituent members

4451 and 4452 of the attachment 445) is displayed through the working environment image and the plurality of index images output to the remote image output device of the remote operation device 20. A plurality of second index points _p21 and _p22 as a result of projecting the plurality of first index points _p11 and _p12 , whose relative positions change with the change in posture, onto the surface of the target object. The operator can grasp each position. As a result, it is possible to improve the recognition accuracy by the operator of the positional relationship between the plurality of parts of the working mechanism 440 and the target object Obj according to the posture change mode.

The plurality of parts may be configured by, for example, one part of the attachment 445 and another part of the arm 443 and/or the boom 441 . For example, the center point of the tip of the attachment 445 and the center point of the tip of the arm 443 (the connection with the attachment 445) may be defined as the first index points p ₁₁ and p ₁₂ , respectively. In this case, the indicator images _M1 and _M2 superimposed on the work environment image are identifiably represented by colors, shapes, patterns, or a combination thereof, so that the indicator images _M1 and _M2 are displayed in real space. The operator can be made aware of the directional arrangement and thus the attitude of the attachment 445 (eg, bucket).

In the above embodiment, the index image M having directivity with respect to the second index point _p2 was output to the remote image output device 221 (see FIG. ₅ ). The index image M having no directivity with respect to may be output to the remote image output device 221 . For example, the index image M is output to the remote image output device 221, in which a two-dimensional figure of a designated shape such as a circle or square centered or centered on the second index point _p2 is arranged in a posture parallel to the horizontal plane. good too. An index image M is output to the remote image output device 221, in which a three-dimensional figure of a designated shape such as a sphere, cube, or polyhedron centered or centered on the second index point _p2 is arranged in a posture parallel to the horizontal plane. good too.

In the remote operation device 20, based on the operation mode of the remote operation mechanism 211 at the first time point t= _t1 recognized by the remote control device 200, the operation mechanism 440 is operated at the first time point t= _t1 or later. A space occupation pattern at two time points t=t ₂ may be estimated or predicted, and a first index point p ₁ may be defined at the working mechanism 440 at the second time point t=t ₂ . The time difference between the first time point t= _t1 and the second time point t= _t2 is set according to the displacement speed of the first index point _p1 according to the change speed of the position and/or posture of the work mechanism 440 in real space. may be

For example, according to the operation mode of the remote control mechanism 211 and/or the actual machine operating mechanism 411, or based on the output signal of the actual machine sensor group 416, the upper rotating body 420 rotates counterclockwise relative to the lower traveling body 410 when viewed from above. Consider the case where turning around is estimated or predicted or measured at a first time point t=t ₁ (see left pointing white arrow in FIG. 13A). In this case, the tip of the bucket 445 at the second time t=t ₂ indicated by the dashed line in FIG. 13A is displaced to the left of the bucket 445 at the first time t=t ₁ indicated by the solid line in FIG. 13A. A central point is defined as the first index point p ₁ .

Further, the following description is based on the operation mode of the remote control mechanism 211 and/or the actual machine operating mechanism 411, or based on the output signal of the actual machine sensor group 416, so that the upper swing body 420 is raised with respect to the lower traveling body 410. Consider the case where a clockwise turn is estimated or predicted or measured at a first time point t=t ₁ (see right pointing white arrow in FIG. 13B). In this case, the tip of the bucket 445 at the second time t=t ₂ indicated by the dashed line in FIG. 13B is displaced to the right of the bucket 445 at the first time t=t ₁ indicated by the solid line in FIG. 13B. A central point is defined as the first index point p ₁ .

In each case, an index image M showing a second index point _p2 as a result of projecting the first index point _p1 onto the surface of the target object Obj is superimposed on the work environment image to form a remote image. It is output to the output device 221 .

According to the image display system having this configuration, communication delay of the working mechanism 440 constituting the working machine 40 can be detected through the working environment image output to the remote image output device 221 of the remote controller 20 and the index image M superimposed thereon. Since the operator can recognize changes in the position and/or posture taking into consideration part or all of the time and response delay time, the positional relationship of the working mechanism 440 with respect to the target object Obj can be determined in consideration of the operating environment unique to remote control. The recognition accuracy of is improved, and operations can be performed more efficiently.

In the image display system of the present invention,
Preferably, the index image having directivity with respect to the second index point is superimposed on the work environment image and output to the output interface of the remote control device.

According to the image display system having this configuration, the index image output to the output interface of the remote control device has directivity with respect to the second index point (the index image indicates the position of the second index point). Therefore, the operator can easily grasp the positional relationship between the first index point of the working mechanism and the second index point on the surface of the target object.

In the image display system of the present invention,
The index image showing the second index point as a result of projecting the first index point onto the surface of the target object in a direction corresponding to the displacement mode or the posture mode of the working mechanism. It is preferable that the image is superimposed on the work environment image and output to the output interface of the remote control device.

According to the image display system having this configuration, the working environment image output to the output interface of the remote control device and the index image superimposed thereon are used to display the direction corresponding to the displacement mode or attitude mode of the working mechanism that constitutes the working machine. , it is possible to improve the recognition accuracy of the operator regarding the positional relationship between the first index point of the working mechanism and the second index point on the surface of the target object in the working environment image.

In the image display system of the present invention,
superimposing the index image showing the second index point as a result of projecting the first index point having a different position according to the displacement mode of the working mechanism onto the surface of the target object, on the working environment image; preferably output to the output interface of the remote control device.

According to the image display system of the present invention, the working environment image output to the output interface of the remote control device and the index image superimposed thereon are displaced to a position corresponding to the displacement mode of the working mechanism constituting the working machine. Further, it is possible to improve the recognition accuracy of the operator regarding the positional relationship between the first index point of the working mechanism and the second index point on the surface of the target object in the working environment image.

In the image display system of the present invention,
After recognizing the position of the first index point of the working mechanism determined based on the output of the sensor mounted on the working machine, the result of projecting the first index point onto the surface of the target object It is preferable that the index image indicating the second index point is superimposed on the working environment image and output to the output interface of the remote control device.

According to the image display system having this configuration, for example, even if it is difficult to recognize the real space position of the first index point by the distance measuring device mounted on the work machine, the index is output to the output interface of the remote control device. Through the image, the operator can easily comprehend or guess the positional relationship between the first index point and the second index point as a result of the projection of the first index point onto the surface of the target object.

In the image display system of the present invention,
Preferably, the first index point is set on the arm top.

According to the image display system having this configuration, when the ground is excavated by the working mechanism, in order to apply a strong force in a series of excavation operations, the position of the tip of the attachment is manipulated with respect to the position of the arm top. As an arm top is suitable. Furthermore, since the position of the arm top does not change even if the attachment is replaced, there is an advantage that the same image display can be applied.

In the image display system of the present invention,
A plurality of index images each showing a plurality of the plurality of second index points as a result of each of the plurality of first index points being projected onto the surface of the target object are superimposed on the work environment image. It is preferable to output to the output interface of the remote control device.

According to the image display system having this configuration, each of the plurality of first index points of the working mechanism is projected onto the surface of the target object through the working environment image and the plurality of index images output to the output interface of the remote control device. As a result, the operator can grasp the position of each of the plurality of second index points. As a result, compared to the case where only a single index image is output to the output interface of the remote control device, it is possible to improve the recognition accuracy by the operator of the positional relationship between the working mechanism and the target object.

In the image display system of the present invention,
a plurality of index images pointing toward each of the plurality of second index points corresponding to each of the plurality of first index points whose relative positions change as the posture of the working mechanism changes; It is preferable that the image is superimposed on the work environment image and output to the output interface of the remote control device.

According to the image display system having this configuration, a plurality of first indices whose relative positions change as the posture of the working mechanism changes through the work environment image and the plurality of index images output to the output interface of the remote control device. The operator can grasp the position of each of the plurality of second index points as a result of each point being projected onto the surface of the target object. As a result, it is possible to improve the recognition accuracy by the operator of the positional relationship between the plurality of parts of the working mechanism and the target object according to the posture change mode.

10... Image display system 20... Remote control device 40... Working machine 41... Actual machine input interface 42... Actual machine output interface 102... Database 121... Communication function element 122... Image processing function element 200... Remote control Device 210 Remote input interface 211 Remote control mechanism 220 Remote output interface 221 Remote image output device 222 Remote sound output device 400 Actual machine control device 410 Lower running body 420 Upper turning body, 424... cab (driver's cab), 440... work mechanism, 445... attachment (breaker, bucket, etc.), M, _M1 , _M2 ... index image, Obj... target object, _p1 , _p11 , _p12 ... First index point, p ₂ , p ₂₁ , p ₂₂ . . . second index point.

Claims

A first index point in the working mechanism is projected onto the surface of the target object in real space on a working environment image representing the situation of the working mechanism constituting the working machine and the target object existing around the working machine. An image display system for superimposing an index image showing a second index point as a result and outputting it to an output interface of a remote control device for remotely controlling the working machine.
In the image display system according to claim 1,
An image display system for superimposing the index image having directivity with respect to the second index point on the work environment image and outputting the image to an output interface of the remote control device.
In the image display system according to claim 1,
The index image showing the second index point as a result of projecting the first index point onto the surface of the target object in a direction corresponding to the displacement mode or the posture mode of the working mechanism. An image display system that superimposes the work environment image and outputs the image to an output interface of the remote control device.
In the image display system according to claim 1,
superimposing the index image showing the second index point as a result of projecting the first index point having a different position according to the displacement mode of the working mechanism onto the surface of the target object, on the working environment image; and an image display system for outputting to the output interface of the remote control device.
In the image display system according to claim 1,
After recognizing the position of the first index point of the working mechanism determined based on the output of the sensor mounted on the working machine, the result of projecting the first index point onto the surface of the target object An image display system for superimposing the index image indicating the second index point on the work environment image and outputting the index image to an output interface of the remote control device.
In the image display system according to claim 1,
The image display system, wherein the first index point is set on the arm top.
In the image display system according to claim 1,
A plurality of index images each showing a plurality of the plurality of second index points as a result of each of the plurality of first index points being projected onto the surface of the target object are superimposed on the work environment image. An image display system that outputs to the output interface of the remote control device.
In the image display system according to claim 7,
a plurality of the index images pointing toward each of the plurality of second index points respectively corresponding to the plurality of first index points whose relative positions change with the operation of the working mechanism; An image display system that superimposes a work environment image and outputs the image to an output interface of the remote control device.
a working machine having a real imaging device, a real ranging device, and a working mechanism;
a remote control device having an output interface for remotely controlling the work machine;
an image display system;
An image display complex system comprising:
The image display system displays, in real space, a working environment image representing the status of a working mechanism that constitutes the working machine and target objects that exist around the working machine acquired through the actual machine imaging device. An output interface of the remote control device by superimposing an index image showing two index points as a result of one index point being projected onto the surface of the target object whose three-dimensional shape is measured by the actual rangefinder. An image display complex system that outputs to
A first index point in the working mechanism is projected onto the surface of the target object in real space on a working environment image representing the situation of the working mechanism constituting the working machine and the target object existing around the working machine. An image display method comprising the step of superimposing an index image showing two index points as a result and outputting it to an output interface of a remote control device for remotely controlling the working machine.