JP7444094B2 - Remote operation support system and remote operation support method - Google Patents

Description

The present invention relates to a technique for remotely controlling a working machine such as a hydraulic excavator using a remote control device.

A technology has been proposed for remote excavators that generates an image of an arc centered on the rotation center axis of a rotating body from distance information determined by a distance detection device, and displays a composite image on a display device by combining it with the image taken by a camera. (For example, see Patent Document 1). Since a two-dimensional image taken with a camera does not give a sense of depth or the unevenness of the terrain, conventional technology superimposes the topographical information acquired by a distance sensor such as LIDAR on the camera image, giving the operator a sense of depth and information about the unevenness of the ground. making it recognizable.

Patent No. 6794193

However, if the image data acquired by the camera and the distance information acquired by LIDAR are combined in a misaligned manner, only information with low usefulness is generated. Therefore, there is a need for a technique that accurately combines image data and distance information.

Therefore, the present invention provides that a captured image acquired through an imaging device mounted on a work machine and distance measurement information included in a distance measurement image acquired through a distance measurement device mounted on the same work machine are The purpose of the present invention is to provide a technical method that can generate an accurately composite environmental image.

The remote operation support server of the present invention includes:
A remote operation support system for supporting remote operation of a work machine using a remote control device,
A marker attached to a ranging device mounted on the working machine and having a ranging range that overlaps an imaging range by the imaging device in real space, acquired through an imaging device mounted on the working machine. a first support processing element that recognizes the relative position and relative orientation of the distance measuring device with respect to the imaging device based on captured images;
Based on the relative position and relative orientation of the distance measuring device with respect to the imaging device recognized by the first support processing element, the distance measuring device outputs to a remote output interface that constitutes the remote control device. a second support processing element that generates an environmental image in which information included in the acquired ranging image is incorporated into the captured image;
It is equipped with

According to the remote operation support system having the above configuration, based on the captured image acquired through the imaging device mounted on the working machine, and furthermore, the reflection state of the marker attached to the distance measuring device also mounted on the working machine. , the relative position and relative orientation of the distance measuring device with respect to the imaging device are recognized. An environment in which distance information included in a distance measurement image is incorporated into a captured image based on the relative position and relative orientation of the distance measurement device with respect to the imaging device, and furthermore, the coordinate transformation mode of the captured image coordinate system and the distance measurement image coordinate system. An image is generated. By outputting this environmental image to the remote output interface that constitutes the remote control device, the operator who is in contact with the environmental image can tell the position (or position and posture) can be accurately recognized.

Further, in the remote operation support system of the present invention, the first support processing element recognizes the relative position of the distance measuring device with respect to the imaging device according to the size of the marker in the captured image.

According to the remote operation support system with this configuration, the relative position of the distance measuring device with respect to the imaging device is recognized according to the size and display position of the marker reflected in the captured image (two-dimensional image). As a result, the relative position of the distance measuring device with respect to the imaging device is recognized based on the reflected appearance of the marker.

Further, in the remote operation support system of the present invention, a plurality of markers are attached to the distance measuring device, and the first support processing element is configured to determine the distance measuring device based on the imaging device according to the plurality of markers. Recognize the relative posture of the device.

According to the remote operation support server having this configuration, the relative attitude of the distance measuring device with respect to the imaging device is recognized according to the plurality of markers reflected in the captured image (two-dimensional image). As a result, the relative attitude of the distance measuring device with respect to the imaging device is recognized based on the reflection mode of the marker.

FIG. 1 is an explanatory diagram regarding the remote operation support system of the present invention. FIG. 2 is an explanatory diagram regarding the configuration of a remote control device. An explanatory diagram regarding the configuration of a working machine. An explanatory diagram regarding the functions of the operation support system. An explanatory diagram regarding a captured image. FIG. 4 is an explanatory diagram regarding the relative positions and orientations of a captured image coordinate system and a ranging image coordinate system. An explanatory diagram regarding an environmental image.

(Configuration of remote operation support system)
The remote operation support complex system shown in FIG. 1 is composed of a remote operation support server 10 (constituting a remote operation support system) and a remote operation device 20 for remotely controlling a work machine 40. There is. The remote operation support server 10, the remote operation device 20, and the work machine 40 are configured to be able to communicate with each other via network. The mutual communication network of the remote operation support server 10 and the remote operation device 20 and the mutual communication network of the remote operation support server 10 and the work machine 40 may be the same or different.

When a component of the present invention "recognizes" information, it means receiving the information, reading the information from a storage device, retrieving the information from a database, measuring the information, receiving the information, etc. Based on the basic information obtained, perform any calculation processing to make the information available for subsequent calculation processing, such as determining, judging, estimating, or predicting the information, or storing the information in a storage device. It is an inclusive concept.

(Configuration of remote operation support server)
The remote operation support server 10 includes a database 102, a first support processing element 121, and a second support processing element 122. The database 102 stores and holds captured image data and the like. The database 102 may be configured by a database server separate from the remote operation support server 10. Each support processing element is composed of an arithmetic processing unit (single-core processor or multi-core processor, or processor cores constituting it), reads necessary data and software from a storage device such as memory, and processes the data and software as target. Therefore, the arithmetic processing described later is executed.

(Configuration of remote control device)
The remote control device 20 includes a remote control device 200, a remote input interface 210, and a remote output interface 220. The remote control device 200 is composed of an arithmetic processing unit (single-core processor or multi-core processor, or processor cores constituting it), reads necessary data and software from a storage device such as memory, and processes the data into the software. Execute the arithmetic processing accordingly.

The remote input interface 210 includes a remote control mechanism 211. The remote output interface 220 includes an image output device 221, an audio output device 222, and a remote wireless communication device 222.

The remote control mechanism 211 includes a travel operating device, a turning operating device, a boom operating device, an arm operating device, and a bucket operating device. Each operating device has an operating lever that is rotated. The operating lever (traveling lever) of the traveling operating device is operated to move the lower traveling body 410 of the work machine 40. The travel lever may also serve as a travel pedal. For example, a travel pedal may be provided that is fixed to the base or lower end of the travel lever. The operating lever (swing lever) of the swing operating device is operated to move a hydraulic swing motor that constitutes the swing mechanism 430 of the working machine 40 . The operating lever (boom lever) of the boom operating device is operated to move the boom cylinder 442 of the work machine 40. The operating lever (arm lever) of the arm operating device is operated to move the arm cylinder 444 of the work machine 40. The operating lever (bucket lever) of the bucket operating device is operated to move the bucket cylinder 446 of the work machine 40.

For example, as shown in FIG. 2, the operating levers constituting the remote control mechanism 211 are arranged around a seat St on which an operator is seated. The seat St is a seating part in any form on which the operator can sit, such as a high-back chair with an armrest, a low-back chair without a headrest, or a chair without a backrest. It may be.

A pair of left and right travel levers 2110 corresponding to left and right crawlers are arranged side by side in front of the seat St. One operating lever may also serve as multiple operating levers. For example, when the left operating lever 2111 provided in front of the left frame of the seat St shown in FIG. 2 functions as an arm lever when operated in the front-back direction, and when operated in the left-right direction It may also function as a pivot lever. Similarly, the right operating lever 2112 provided in front of the right frame of the seat St shown in FIG. It may function as a bucket lever in some cases. The lever pattern may be arbitrarily changed according to an operator's operation instruction.

For example, as shown in FIG. 2, the image output device 221 includes a central image output device 2210 having substantially rectangular screens disposed in front of the seat St, diagonally to the left, and diagonally to the right, and a left side image. It is composed of an output device 2211 and a right-side image output device 2212. The shapes and sizes of the screens (image display areas) of the central image output device 2210, left side image output device 2211, and right side image output device 2212 may be the same or different.

As shown in FIG. 2, the left image output device 2211 The right edge of is adjacent to the left edge of central image output device 2210 . As shown in FIG. 2, the right image output device 2212 is arranged such that the screen of the center image output device 2210 and the screen of the right image output device 2212 form an inclination angle δ2 (for example, 120°≦δ2≦150°). The left edge of is adjacent to the right edge of central image output device 2210 . The inclination angles δ1 and δ2 may be the same or different.

The screens of the central image output device 2210, the left image output device 2211, and the right image output device 2212 may be parallel to the vertical direction or may be inclined to the vertical direction. At least one of the central image output device 2210, the left image output device 2211, and the right image output device 2212 may be configured by a plurality of divided image output devices. For example, the central image output device 2210 may include a pair of vertically adjacent image output devices each having a substantially rectangular screen.

The number of image output devices constituting the image output device 221 may be changed arbitrarily. For example, the image output device 221 may be configured as one image output device having a curved surface shape or a bent surface shape so as to surround the sheet St at the front. The image output device 221 may be constituted by four or more planar image output devices arranged in succession in the lateral direction so as to surround the sheet St in the front.

The sound output device 222 is composed of a speaker, and includes a central sound output device 2220 and a left sound output device arranged at the rear of the seat St, the rear of the left armrest, and the rear of the right armrest, respectively, as shown in FIG. 2, for example. 2221 and a right side sound output device 2222. The specifications of the central sound output device 2220, the left side sound output device 2221, and the right side sound output device 2222 may be the same or different.

(Composition of working machine)
The work machine 40 includes a real machine control device 400, a real machine input interface 41, a real machine output interface 42, and a work mechanism 440. The actual machine control device 400 is composed of an arithmetic processing device (a single-core processor or a multi-core processor, or a processor core constituting this processor), reads necessary data and software from a storage device such as a memory, and processes the data into the software. Execute the arithmetic processing accordingly.

The work machine 40 is, for example, a crawler excavator (construction machine), and as shown in FIG. and an upper revolving body 420. A cab 424 (operator's cab) is provided at the front left side of the upper revolving body 420. A working mechanism 440 is provided at the front central portion of the upper revolving body 420.

The real device input interface 41 includes a real device operating mechanism 411, a real device imaging device S1, and a distance measuring device S2. The work machine 40 may include a positioning device including a GPS and, if necessary, a gyro sensor or the like. The actual machine operating mechanism 411 includes a plurality of operating levers arranged around a seat arranged inside the cab 424 in the same way as the remote operating mechanism 211 . The cab 424 is provided with a drive mechanism or a robot that receives a signal corresponding to the operating mode of the remote control lever and moves the actual control lever based on the received signal. The actual machine imaging device S1 is arranged in the interior space of the cab 424, and images the working environment including at least a portion of the working mechanism 440 or the surroundings of the working machine 40 through the front window and the pair of left and right side windows. Some or all of the front window and side windows may be omitted. The distance measuring device S2 is arranged in the interior space of the cab 424 and measures the distance to objects such as the ground around the work machine 40 through the front window.

The actual imaging device S1 is arranged at the rear of the interior space of the cab 424 (for example, near or behind the headrest of the seat). Therefore, as shown in FIG. 5, the captured image acquired through the imaging device S1 includes the operating lever and part of the seat that constitute the actual machine operating mechanism 411 arranged in the interior space of the cab 424. , the interior space of the cab 424 is reflected. The distance measuring device S2 is arranged in the interior space of the cab 424 at a position included in the imaging range of the actual imaging device S1 (for example, at the front of the interior space of the cab 424, or in a position ahead of the imaging device S1 or closer to the front window). ing.

The position and orientation of the actual imaging device S1 may be fixed with respect to the working machine 40, but the position and/or orientation of the actual imaging device S1 may be adjusted to the lower side with respect to the working machine 40 by a drive mechanism. .

Therefore, as shown in FIG. 5, the distance measuring device S2 and the three markers M0, M1, and M2 attached to the distance measuring device S2 are reflected in the captured image acquired through the imaging device S1. I'm here.
The marker M0 is arranged, for example, on the upper back side of the housing of the distance measuring device S2. The markers M1 and M2 are arranged, for example, at the lower part of the back of the housing of the range finder S2 or on the back of a frame or bracket that supports the housing, spaced apart laterally. The average coordinate values of markers M0, M1, and M2 are located on the back side of distance measuring device S2. Further, any one of the three markers M0, M1, and M2 (for example, M0) may be arranged on the back side of the distance measuring device S2.

The real device output interface 42 includes a real device wireless communication device 422.

As shown in FIG. 3, the working mechanism 440 as a working mechanism includes a boom 441 that is attached to the upper revolving body 420 so as to be able to raise and lower, and an arm 443 that is rotatably connected to the tip of the boom 441. and a bucket 445 rotatably connected to the tip of the arm 443. The working mechanism 440 is equipped with a boom cylinder 442, an arm cylinder 444, and a bucket cylinder 446, each of which is an extendable hydraulic cylinder.

The boom cylinder 442 is interposed between the boom 441 and the upper revolving structure 420 so as to expand and contract when supplied with hydraulic oil and rotate the boom 441 in the up-and-down direction. The arm cylinder 444 is interposed between the arm 443 and the boom 441 so as to expand and contract when supplied with hydraulic oil and rotate the arm 443 around a horizontal axis with respect to the boom 441 . The bucket cylinder 446 is interposed between the bucket 445 and the arm 443 so as to expand and contract when supplied with hydraulic oil and rotate the bucket 445 about a horizontal axis relative to the arm 443 .

(function)
The functions of the remote operation support system having the above configuration will be explained using the flowchart shown in FIG. In the flowchart, the block "C●" is used to simplify the description, and means the transmission and/or reception of data, and the processing in the branch direction is executed on the condition that the data is transmitted and/or received. It means conditional branching. The received data is stored in a storage device comprising a database 102 and/or non-volatile or volatile memory.

In the remote control device 20, it is determined whether the operator has performed a designated operation through the remote input interface 210 (FIG. 4/STEP 210). The "designation operation" is, for example, an operation such as a tap on the remote input interface 210 for designating the work machine 40 that the operator intends to remotely control. If the determination result is negative (FIG. 4/STEP 210...NO), the series of processes ends. On the other hand, if the determination result is positive (FIG. 4/STEP 210...YES), an environment confirmation request is sent to the remote operation support server 10 via the remote wireless communication device 222 (FIG. 4/STEP 212).

When the environment confirmation request is received in the remote operation support server 10, the first support processing element 121 transmits the environment confirmation request to the corresponding work machine 40 (FIG. 4/C110).

In the work machine 40, when an environment confirmation request is received through the actual machine wireless communication device 422 (FIG. 4/C410), the actual machine control device 400 acquires a captured image through the actual machine imaging device S1, and performs measurement through the distance measuring device S2. A distance image is acquired (FIG. 4/STEP 410). Here, image processing for reducing the amount of image data and/or removing noise is performed on the captured image and/or the distance measurement image by the actual device control device 400 or an image processing device constituting the same. may be done. The actual device control device 400 transmits captured image data corresponding to the captured image and ranging image data corresponding to the ranging image to the remote control server 10 via the actual wireless communication device 422 (FIG. 4/STEP 412).

When the remote operation support server 10 receives the captured image data and the ranging image data (FIG. 4/C112), as shown in FIG. Based on the captured image containing M2, the first support processing element 121 recognizes the relative position and relative orientation of the distance measuring device S2 with respect to the actual imaging device S1 (FIG. 4/STEP 111). This is a coordinate transformation matrix between the imaging device coordinate system C(S1) = (X1, Y1, Z1) and the distance measuring device coordinate system C(S2) = (X2, Y2, Z2) shown in FIG. Or, it is synonymous with finding the equivalent quaternion. The X1 axis of the imaging device coordinate system C (S1) is defined to coincide with the optical axis of the imaging device S1. The Y1-Z1 plane coordinate system constitutes the captured image coordinate system. The X2 axis of the range finder coordinate system C(S2) is defined to coincide with the optical axis of the range finder S2. The Y2-Z2 plane coordinate system constitutes the ranging image coordinate system.

Determining the relative position of the range finder S2 with respect to the actual image capture device S1 is performed by moving from the origin O1 of the image capture device coordinate system C(S1) to the origin of the range finder coordinate system C(S2) shown in FIG. This is equivalent to finding the vector R extending to O2. For example, based on the sizes of markers M0, M1, and M2 in the captured image shown in FIG. The length (norm) of each is determined. A geometric relationship is used in which the larger the size of the markers M0, M1, and M2, the shorter the length of each marker from the origin O1. Furthermore, from the length determined in the previous step and the display positions of the markers M0, M1, and M2 on the captured image shown in FIG. 5, the imaging device coordinate system C (S1 ) are determined. In the imaging device coordinate system C (S1), the more the marker shifts from an arbitrary point on the X1 axis in the Y1 axis direction and the X1 axis direction, the longer the length determined in the previous process becomes (the size of the marker on the captured image becomes smaller). This geometric relationship is used.

Furthermore, a vector R in the imaging device coordinate system C (S1) shown in FIG. 6 is determined based on the average coordinate values of the markers M0, M1, and M2 in the captured image. The vector R can also be expressed by the length of the vector R, the azimuth angle θ1, and the elevation angle φ1. Vector R1 shown in FIG. 6 schematically represents the result of projection of vector R onto the X1-Y1 plane, and the angle that vector R1 makes with respect to the X1 axis corresponds to the azimuth angle θ1. Similarly, vector R2 schematically represents the result of projection of vector R onto the Y1-Z1 plane, and the angle that vector R2 makes with respect to the Y1 axis corresponds to the elevation angle φ1. Based on the length of the vector R, the azimuth angle θ1, and the elevation angle φ1, a translation matrix between the imaging device coordinate system C (S1) and the ranging device coordinate system C (S2) is determined. Furthermore, if any one of the three markers M0, M1, and M2 (for example, M0) is located on the back of the distance measuring device S2, the vector extending from the origin O1 to the marker M0 is Corresponds to vector R. Further, the vector R may be determined in consideration of the relative positional relationship with respect to the three markers M0, M1, and M2.

Determining the relative attitude of the distance measuring device S2 with respect to the actual imaging device S1 is performed by changing the imaging device coordinate system C (S1) shown in FIG. 6 from the origin O1 to the distance measuring device coordinate system C (S2) This is equivalent to determining the azimuth angle θ2 and the elevation angle φ2 of the rangefinder coordinate system C (S2) based on the coordinate system C (S1) that has been translated so as to coincide with the origin O2 of the range finder coordinate system C (S2). For example, since marker M0, marker M1, and marker M2 are located on the Y2-Z2 plane, the Y2-Z2 plane is specified from the coordinates of the three markers, and the X2 axis is perpendicular to the Y2-Z2 plane or the azimuth angle from the vertical. The axis in the direction offset from θ2 and the elevation angle φ2 and passing through the origin O2 is specified as the X2 axis. The offset amount is set by checking in advance the amount of inclination of the optical axis when the distance measuring device S2 to which the three markers M0, M1, and M2 are attached is set on a specific plane.

Further, as another method for specifying the X2 axis, in the captured image shown in FIG. 5, when a plurality of markers, for example, marker M1 and marker M2 are arranged side by side, Based on the interval, the azimuth angle θ2 shown in FIG. 6 is determined. A geometric relationship is used in which the smaller the interval, the larger the azimuth angle θ2. The elevation angle φ2 shown in FIG. 6 is determined based on the interval or average interval between the marker M0 and one or both of the markers M1 and M2 in the captured image shown in FIG. A geometric relationship is used in which the smaller the interval, the larger the elevation angle φ2. Based on the azimuth angle θ2 and the elevation angle φ2, a rotation matrix between the imaging device coordinate system C (S1) and the ranging device coordinate system C (S2) is determined.

If the positions and postures of the actual imaging device S1 and the distance measuring device S2 in the internal space of the work machine 40 or the cab 424 remain unchanged, the process in STEP 111 may be executed only in the initial stage and may be omitted thereafter.

Based on the relative position and relative orientation of the distance measuring device S2 with respect to the actual imaging device S1 recognized by the first support processing element 121, the second support processing element 122 determines whether the distance measurement device is included in the captured image. An environment image in which the distance information is combined is generated (FIG. 4/STEP 112). Specifically, the distance to the object, which is the pixel value of each pixel in the ranging image coordinate system (Y2, Z2), is assigned to each pixel in the captured image coordinate system (Y1, Z1). At this time, the coordinate values of each pixel in the ranging image coordinate system (Y2, Z2) are changed to the captured image coordinate system (Y1, Z1) by using the translation matrix and rotation matrix obtained as described above. Coordinates are converted to coordinate values. Due to the difference in resolution between the actual imaging device S1 and the distance measuring device S2, the coordinate values of each pixel in the ranging image coordinate system (Y2, Z2) after the coordinate transformation are generally the same as the captured image coordinate system ( Y1, Z1) do not match the coordinate values of each pixel. Therefore, according to an appropriate interpolation method, the pixel value of each pixel in the ranging image coordinate system (Y2, Z2) is changed to each pixel (not necessarily all pixels) in the captured image coordinate system (Y1, Z1). assigned to. As a result, the real space position of each of a plurality of locations of the object reflected in the captured image with reference to the imaging device S1, and the shape of the surface irregularities of the object are estimated.

As a result, for example, as shown in FIG. 7, a mesh-like line diagram is created on the ground as an object included in the captured image according to the distance information included in the ranging image. An environment image with superimposed images is generated. The extension mode of the latitude map in the mesh-like line diagram is the pixel value (distance measurement The distance from the device S2 to the object) is determined according to the change in the distance from the device S2 to the object. The extension mode of the meridian diagram in the mesh-like line diagram is the pixel value (distance measurement The distance from the device S2 to the object) is determined according to the change in the distance from the device S2 to the object.

The second support processing element 122 transmits environmental image data corresponding to the captured image to the remote control device 20 (FIG. 4/STEP 114).

When the remote control device 20 receives environmental image data through the remote wireless communication device 222 (FIG. 4/C210), the remote control device 200 outputs an environmental image according to the environmental image data to the image output device 221. (Figure 4/STEP 214). As a result, an image as shown in FIG. 7, such as a crowded environment image shown through the front window of the cab 424, is output to the image output device 221.

In the remote control device 20, the operation mode of the remote control mechanism 211 is recognized by the remote control device 200 (FIG. 4/STEP 216), and a remote control command corresponding to the operation mode is sent to remote control support through the remote wireless communication device 222. It is transmitted to the server 10 (FIG. 4/STEP 218).

In the remote operation support server 10, when the second support processing element 122 receives the remote operation command, the first support processing element 121 transmits the remote operation command to the work machine 40 (FIG. 4/ C114).

In the work machine 40, when the actual machine control device 400 receives an operation command through the real machine wireless communication device 422 (FIG. 4/C412), the operation of the work mechanism 440, etc. is controlled (FIG. 4/STEP 414). For example, the soil in front of the working machine 40 is scooped up using the bucket 445, the upper revolving body 410 is rotated, and then the soil is dropped from the bucket 445.

(effect)
According to the remote operation support system having this configuration, the captured image acquired through the imaging device S1 mounted on the working machine 40, and furthermore, the marker M0 attached to the distance measuring device S2 also mounted on the working machine 40, Based on the reflection mode of M1 and M2, the relative position and relative orientation of the distance measuring device S2 with respect to the imaging device S1 are recognized (see FIG. 4/STEP 111, FIGS. 5 and 6). The distance information included in the ranging image is incorporated into the captured image based on the relative position and relative orientation of the ranging device S2 with respect to the imaging device S1, as well as the coordinate transformation mode of the captured image coordinate system and the ranging image coordinate system. An environmental image is generated (see FIG. 4/STEP 112 and FIG. 7).

By outputting this environmental image to the remote output interface 220 that constitutes the remote control device 20, the operator who is in contact with the environmental image can see the position in the three-dimensional real space of the object reflected in the captured image (two-dimensional image). (or position and posture) can be accurately recognized.

(Other embodiments of the present invention)
At least some functional elements of the remote support processing server 10 may be configured by the remote control device 20 and/or the work machine 40. For example, the first support processing element 121 may be configured by the remote control device 200 and/or the real machine control device 400 as the first arithmetic processing device. The second support processing element 122 may be configured by a remote control device 200 and/or a real machine control device 400 as a second arithmetic processing device. When the functional elements of the remote operation support server 10 are installed in the remote operation device 20, information is communicated by wired communication through a wired network installed in the remote operation device 20 instead of the wireless communication in the above embodiment. may be done. Similarly, when the functional elements of the remote operation support server 10 are installed in a work machine 40, information is transmitted by wired communication through a wired network installed in the work machine 40, instead of the wireless communication in the above embodiment. may be communicated.

For example, the functional elements represented by STEP112 and STEP114 in FIG. 4 or C112, STEP111, STEP112, and STEP114 in FIG. 4 may be configured by the remote control device 200. The functional elements represented by C112 and STEP 111 in FIG. 4 or C112, STEP 111, STEP 112, and STEP 1114 in FIG.

The number of markers may be four or more. The plurality of markers M0, M1 and M2 may be congruent, similar or dissimilar. The shape of each of the plurality of markers M0, M1, and M2 may be selected from various shapes such as a circle, an ellipse, a triangle, a square, a rectangle, a trapezoid, a polygon such as a regular hexagon, and a star shape. Alternatively, a two-dimensional code (QR code (registered trademark)) may be assigned to the marker, and the marker may be identified by extracting the two-dimensional code from the captured image. When the orientation of the distance measuring device coordinate system C (S2) in the imaging device coordinate system C (S1) shown in FIG. Good too. If the azimuth angle θ2 or the elevation angle φ2 of the distance measuring device coordinate system C (S2) in the imaging device coordinate system C (S1) shown in FIG. 6 is known, the number of markers may be 2, and the elevation angle In order to determine φ2 or azimuth θ2 from the geometrical relationship as described above, the two markers may be spaced apart in the vertical or horizontal direction. Two markers may be attached to the distance measuring device S2 so as to be disposed shifted in each of the vertical and horizontal directions.

The relative attitude of the distance measuring device S2 with respect to the imaging device S1 may include an element of twist. For example, if the relative display positions of the markers M0, M1, and M2 on the captured image shown in FIG. Or it is installed counterclockwise. The first support processing element 121 may recognize a relative posture that takes such twisting into consideration.

In the embodiment, the average coordinate value of the markers M0, M1, M2 or any one of the three markers M0, M1, M2 (for example, M0) is arranged to be located on the back side of the distance measuring device S2. However, as another embodiment, it may be set by confirming in advance the relative position of the back surface of the distance measuring device S2 with respect to the markers M0, M1, and M2. The degree of freedom in arranging the three markers M0, M1, and M2 is improved.

In the embodiment described above, both the imaging device S1 and the distance measuring device S2 are arranged in the interior space of the cab 424 in the working machine 40, but in another embodiment, the distance measuring device S2 is attached to the imaging range of the imaging device S1. The working machine 40 uses the imaging device S1 on the condition that the markers M0, M1, and M2 are reflected, and the imaging range of the imaging device S1 and the ranging range of the ranging device S2 overlap in real space. and the distance measuring device S2 may be arranged in different forms. For example, the imaging device S1 may be disposed in the interior space of the cab 424 facing forward, while the distance measuring device S2 may be disposed in front of the cab 424 facing forward. .

10...Remote operation support server, 20...Remote operation device, 40...Work machine, 102...Database, 121...First support processing element, 122...Second support processing element, 200...Remote control device, 210...Remote input interface, 211... Remote control mechanism, 220... Remote output interface, 221... Image output device, 222... Sound output device, 400... Actual machine control device, 410... Actual machine input interface, 420... Actual machine output interface, 424... Cab (driver's cab), 440... Working mechanism, 445... Bucket (working part), S1... Actual imaging device, S2... Distance measuring device, M0, M1, M2... Marker.

Claims (4)

A remote operation support system for supporting remote operation of a working machine using a remote control device,
A marker attached to a ranging device mounted on the working machine and having a ranging range that overlaps an imaging range by the imaging device in real space, acquired through an imaging device mounted on the working machine. a first support processing element that recognizes the relative position and relative orientation of the distance measuring device with respect to the imaging device based on captured images;
Based on the relative position and relative orientation of the range finder with respect to the imaging device recognized by the first support processing element, the range finder outputs the information to a remote output interface constituting the remote control device. a second support processing element that generates an environmental image in which information included in the acquired ranging image is incorporated into the captured image;
A remote control support system equipped with The remote operation support system according to claim 1,
The first support processing element recognizes the relative position of the distance measuring device with respect to the imaging device according to the size and display position of the marker in the captured image. The remote operation support system according to claim 1 or 2,
A plurality of markers are attached to the distance measuring device,
The first support processing element recognizes a relative attitude of the distance measuring device with respect to the imaging device according to the plurality of markers. A remote operation support method for supporting remote operation of a working machine using a remote control device,
A marker attached to a ranging device mounted on the working machine and having a ranging range that overlaps an imaging range by the imaging device in real space, acquired through an imaging device mounted on the working machine. a first support processing step of recognizing the relative position and relative orientation of the distance measuring device with respect to the imaging device based on captured images;
Based on the relative position and relative orientation of the distance measuring device with respect to the imaging device recognized in the first support processing step, the distance measuring device outputs to a remote output interface that constitutes the remote control device. a second support processing step of generating an environmental image in which information included in the acquired ranging image is incorporated into the captured image;
A remote operation support method including.

Images