JP2024057362A - Work support system, remote control support system, work support method, and display device - Google Patents

Abstract

To provide a system and the like capable of improving the recognition accuracy of the space occupancy of the tip of the working mechanism; i.e. what will happen when an object that is hidden underground and has not appeared, appears on the surface in the future in a work environment image that represents the surrounding situation of a working machine.SOLUTION: A work support image in which a first index image and a second index image are superimposed on a work environment image is generated. The "work environment image" is an image that represents the surrounding situation of working machine 40 acquired through an actual imaging device 412; especially, an image showing the situation where the cutting edge E of a bucket 445 (the tip of the working mechanism 44) is hidden underground. The "first index image" is an image indicating a projection result to the ground surface of the space occupancy mode of the cutting edge E when the cutting edge E of the bucket 445 predicted depending on the working or operating mode of the working mechanism 44 appears on the ground.SELECTED DRAWING: Figure 6

Description

The present invention relates to technology for assisting operators in remotely controlling work machines such as hydraulic excavators.

A technology has been proposed for generating a display image in which a captured image of a working mechanism captured by a camera mounted on a working machine, a cutting edge shadow projected vertically onto a projection surface of the working mechanism's cutting edge, and a reference range figure projected vertically onto the projection surface to show the reachable range of the cutting edge (see, for example, Patent Document 1). This aims to prevent a decrease in work efficiency when working using a working machine having a working mechanism.

JP 2022-055808 A

However, when the tip of a bucket or other component of a working mechanism is at least partially hidden underground, it is unclear how the tip will appear above ground, which may hinder the operator's operation of the working mechanism. This may result in a decrease in the work efficiency of the work machine.

The present invention aims to provide a system that can improve the accuracy of recognizing the spatial occupation pattern of the tip of a work mechanism when it appears on the ground surface in the future, even if it is hidden underground and not visible in a work environment image showing the situation around the work machine.

The work support system of the present invention comprises:
A work support image is generated by superimposing a first index image representing the projection result onto the ground surface of the spatial occupation mode of the tip of the work mechanism when the tip of the work mechanism emerges above the ground surface, which is predicted according to the operation mode or operating mode of the work mechanism, on a work environment image representing a situation in which the tip of the work mechanism is hidden underground, which is acquired by an imaging device around a work machine having a work mechanism.

According to the work support system having this configuration, a work support image is generated in which a first index image is superimposed on a work environment image. The "work environment image" is an image captured by an imaging device that shows the situation around the work machine, in particular an image showing a situation in which the tip of the work mechanism is hidden underground. The "first index image" is an image that shows the projection onto the ground surface of the spatial occupation mode of the tip of the work mechanism when the tip of the work mechanism appears on the ground surface, which is predicted according to the operating mode or operation mode of the work mechanism.

The "space occupancy mode" of an object is specified, for example, by the coordinate values of at least a part of a point cloud that constitutes the space that the object occupies in real space and/or virtual space. The "ground surface" is a concept that includes curved and/or uneven surfaces that reproduce with high accuracy the shape of the ground surface around the work machine and reflected in the work environment image, as well as flat or horizontal surfaces that roughly follow the ground surface.

Therefore, even if the work mechanism is hidden underground (or not reflected) in the work environment image, the operator or the like who comes into contact with the work support image can recognize the projection result onto the ground surface of the spatial occupation mode of the tip of the work mechanism when it appears on the ground surface, which is predicted in consideration of the operating mode of the work mechanism.

In the work assistance system having the above configuration,
It is preferable to generate a work support image in which the first indicator image is superimposed on the work environment image, the first indicator image serving as a pointer pointing to the projection onto the ground surface of one or more points constituting the contour or tip edge of the tip of the work mechanism when the tip of the work mechanism is predicted to appear on the ground surface according to the operation mode or operation mode of the work mechanism, or the projection onto the ground surface of one or more points constituting the contour or tip edge of the tip of the work mechanism when the tip of the work mechanism is predicted to appear on the ground surface according to the operation mode or operation mode of the work mechanism.

According to the work support system of this configuration, although the work mechanism is hidden underground in the work environment image, the projection result of the outline or tip edge of the tip of the work mechanism onto the ground surface when it appears on the ground surface, which is predicted in consideration of the operating mode of the work mechanism, can be made to be recognized by an operator or the like who comes into contact with the work support image that includes the line drawing or the first indicator image as the pointer.

In the work assistance system having the above configuration,
It is preferable to generate the work support image including a second index image that represents or points to the projection result, onto the ground surface, of the spatial occupancy manner of the tip part of the work mechanism hidden underground.

The work support system of this configuration can allow an operator or the like who comes into contact with the work support image to recognize the projection results on the ground surface of the spatial occupation pattern of the tip of the work mechanism hidden underground in the work environment image.

In the work assistance system having the above configuration,
It is preferable to generate a work support image in which the second indicator image is superimposed on the work environment image as a line diagram that is the projection onto the ground surface of the contour or tip edge of the tip of the work mechanism that is hidden underground, or as a pointer that points to the projection onto the ground surface of one or more points that constitute the contour or tip edge of the tip of the work mechanism that is hidden underground.

The work support system of this configuration allows an operator or the like who comes into contact with a work support image that includes the line drawing or the second indicator image as the pointer to recognize the outline or tip edge of the tip of the work mechanism hidden underground in the work environment image, or the projection result of a part of these onto the ground surface.

In the work assistance system having the above configuration,
It is preferable to generate the work support image in which the first index image, which represents the projection result onto the ground surface of the time series of the spatial occupation mode of the tip of the work mechanism from underground until the tip of the work mechanism emerges from the ground on the surface, predicted based on the motion mode or operation mode of the work mechanism, is superimposed on the work environment image.

With this configuration of work support system, even if the tip of the work mechanism is hidden underground in the work environment image, the time series of the spatial occupation state of the tip of the work mechanism until it appears on the ground surface, which is predicted in consideration of the operating state of the work mechanism, can be projected onto the ground surface, allowing an operator or the like who comes into contact with the work support image to recognize it.

In the work assistance system having the above configuration,
It is preferable to generate the work support image in which the first indicator image, which is a figure representing a closed surface resulting from the time-series projection of the tip edge of the tip of the work mechanism onto the ground surface, is superimposed on the work environment image.

According to the work support system of this configuration, although the tip edge of the tip of the work mechanism is hidden underground in the work environment image, the time series of projection results of the tip edge of the tip of the work mechanism onto the ground surface until it appears on the ground surface, which is predicted in consideration of the operating mode of the work mechanism, can be recognized by an operator or the like who comes into contact with the work support image that includes the first indicator image as a closed surface. A closed surface is a concept that encompasses a closed plane.

In the work assistance system having the above configuration,
It is preferable to generate the work support image in which a line drawing that is tangent to the closed surface as the first index image and extends along the ground surface in the operating direction of the work mechanism is further superimposed on the work environment image.

The work support system of this configuration can allow an operator or the like who comes across a work support image that includes the line diagram to recognize the potential spatial occupation pattern of the tip of a work machine hidden underground in the work environment image, depending on the operation of the work mechanism.

FIG. 2 is an explanatory diagram regarding the configuration of a work assistance device. FIG. 2 is an explanatory diagram relating to the configuration of a remote control device. FIG. 2 is an explanatory diagram relating to the configuration of a work machine. FIG. 2 is an explanatory diagram relating to functions of the work support device of the first embodiment. FIG. 4 is an explanatory diagram regarding a space occupation mode of a working mechanism. FIG. 11 is an explanatory diagram relating to a work support image (first embodiment). FIG. 11 is an explanatory diagram relating to a work support image (second embodiment); FIG. 13 is an explanatory diagram relating to a task support image (third embodiment). FIG. 13 is an explanatory diagram relating to a work support image (fourth embodiment); FIG. 13 is an explanatory diagram relating to a task support image (fifth embodiment); FIG. 11 is an explanatory diagram relating to functions of a work support device according to a second embodiment.

(Configuration of the work support device)
1 is composed of a work support device 10 as one embodiment of the present invention, and a remote control device 20 and/or a work machine 40 configured to be able to communicate with the work support device 10 via a network. The mutual communication network between the work support device 10 and the remote control device 20 and the mutual communication network between the work support device 10 and the work machine 40 may be the same as or different from each other.

(Configuration of the work support device)
The work support device 10 is configured with one or more computers or server computers. As shown in Fig. 1, the work support device 10 includes a database 102, a work mechanism space occupation mode recognition element 122, and a work support image generation element 124.

The database 102 stores and holds captured image data as well as size information of the working mechanism 44. The database 102 may be configured as a database server or device that is configured to be able to communicate with the work support device 10 via a network. Each component of the work support device 10 is configured as a calculation processing device (a single-core processor or a multi-core processor, or a processor core that constitutes the same), reads necessary data and software from a storage device such as a memory, and executes the calculation processing described below according to the software on the data.

The components of the present invention "recognize" information (or data), which encompasses any process that prepares the information in a form that can be used to perform subsequent computations, such as obtaining the information by receiving, reading, searching, etc., or determining, measuring, identifying, estimating, predicting, etc. the information by performing computations on the underlying data or signals.

(Configuration of remote control device)
1, the remote operation device 20 includes a remote control device 200, a remote input interface 210, and a remote output interface 220. The remote control device 200 is configured with an arithmetic processing device (a single-core processor or a multi-core processor or a processor core constituting the same), reads necessary data and software from a storage device such as a memory, and executes arithmetic processing on the data according to the software.

The remote input interface 210 includes a remote operation mechanism 211. The remote output interface 220 includes a remote image output device 221, a remote audio output device 222, and a remote wireless communication device 224.

The remote control mechanism 211 includes a travel operation device, a slewing operation device, a boom operation device, an arm operation device, and a bucket operation device. Each operation device has an operation lever that receives a rotation operation. The operation lever (travel lever) of the travel operation device is operated to move the lower travel body 41 of the work machine 40. The travel lever may also serve as a travel pedal. For example, a travel pedal fixed to the base or lower end of the travel lever may be provided. The operation lever (slewing lever) of the slewing operation device is operated to move the hydraulic slewing motor that constitutes the slewing mechanism 43 of the work machine 40. The operation lever (boom lever) of the boom operation device is operated to move the boom cylinder 442 of the work machine 40. The operation lever (arm lever) of the arm operation device is operated to move the arm cylinder 444 of the work machine 40. The operation lever (bucket lever) of the bucket operation device is operated to move the bucket cylinder 446 of the work machine 40.

The operating levers constituting the remote control mechanism 211 are arranged around the seat St on which the operator sits, as shown in FIG. 2, for example. The seat St is in the form of a high-back chair with armrests, but it may be in the form of a low-back chair without a headrest, or in the form of a chair without a backrest, or any other form of seating on which the operator can sit.

Left and right travel levers 2110 corresponding to the left and right crawlers are arranged side by side in front of the seat St. One operation lever may serve as multiple operation levers. For example, the left operation lever 2111 provided in front of the left frame of the seat St shown in FIG. 2 may function as an arm lever when operated in the forward/backward direction, and as a rotation lever when operated in the left/right direction. Similarly, the right operation lever 2112 provided in front of the right frame of the seat St shown in FIG. 2 may function as a boom lever when operated in the forward/backward direction, and as a bucket lever when operated in the left/right direction. The lever pattern may be changed arbitrarily in response to a predetermined operation of the remote input interface 210 by the operator.

The remote image output device 221 (corresponding to the "display device" of the present invention) is composed of a central remote image output device 2210, a left remote image output device 2211, and a right remote image output device 2212, each having a substantially rectangular screen, which are arranged in front of the seat St, diagonally forward to the left, and diagonally forward to the right, as shown in FIG. 2 for example. The screens (image display areas) of the central remote image output device 2210, the left remote image output device 2211, and the right remote image output device 2212 may have the same shape and size or may be different. The remote image output device 221 may be composed of a single curved or bendable image output device, or two or four or more image output devices arranged to surround the front of the seat St. The remote image output device 221 may be composed of an image output device of an information terminal device carried by an operator who operates the remote operation device 20.

As shown in FIG. 2, the right edge of the left remote image output device 2211 is adjacent to the left edge of the central remote image output device 2210 such that the screens of the central remote image output device 2210 and the left remote image output device 2211 form an inclination angle θ1 (e.g., 120°≦θ1≦150°). As shown in FIG. 2, the left edge of the right remote image output device 2212 is adjacent to the right edge of the central remote image output device 2210 such that the screens of the central remote image output device 2210 and the right remote image output device 2212 form an inclination angle θ2 (e.g., 120°≦θ2≦150°). The inclination angles θ1 and θ2 may be the same or different.

The screens of the central remote image output device 2210, the left remote image output device 2211, and the right remote image output device 2212 may be parallel to the vertical direction or inclined to the vertical direction. At least one of the central remote image output device 2210, the left remote image output device 2211, and the right remote image output device 2212 may be composed of multiple divided image output devices. For example, the central remote image output device 2210 may be composed of adjacent image output devices above and below, each having a roughly rectangular screen.

The remote audio output device 222 is composed of one or more speakers, and is composed of a central audio output device 2220, a left audio output device 2221, and a right audio output device 2222, which are arranged behind the seat St, at the rear of the left armrest, and at the rear of the right armrest, as shown in FIG. 2. The specifications of the central audio output device 2220, the left audio output device 2221, and the right audio output device 2222 may be the same or different.

(Configuration of the work machine)
1, the work machine 40 is equipped with an actual machine control device 400, an actual machine input interface 410, an actual machine output interface 420, and an actual machine wireless communication device 422. Each of the components of the actual machine control device 400 is configured by an arithmetic processing device (a single-core processor or a multi-core processor or a processor core constituting the same), reads necessary data and software from a storage device such as a memory, and executes arithmetic processing on the data in accordance with the software.

The work machine 40 is, for example, a crawler excavator (construction machine), and as shown in Fig. 3, includes a crawler-type lower traveling body 41 and an upper rotating body 42 that is rotatably mounted on the lower traveling body 41 via a rotating mechanism 43. A cab 42C (operator's compartment) is provided on the front left side of the upper rotating body 42. A working mechanism 44 is provided in the front center of the upper rotating body 42.

The actual machine input interface 410 includes an actual machine operation mechanism 411, an actual machine imaging device 412, an actual machine positioning device 414, and an actual machine attitude sensor 416.

The actual machine operation mechanism 411 has multiple operation levers arranged around a seat arranged inside the cab 42C in the same manner as the remote operation mechanism 211. A drive mechanism or robot that receives a signal corresponding to the operation mode of the remote operation lever and moves the actual machine operation lever based on the received signal is provided in the cab 42C. The actual machine imaging device 412 is installed, for example, inside the cab 42C, and captures an image of the environment including at least a part of the working mechanism 44 through the front window and the left and right side windows. Some or all of the front window and side windows may be omitted.

The actual machine positioning device 414 is configured with a GPS or GNSS and, if necessary, a gyro sensor, etc., and measures the actual space position of the work machine 40 (for example, the actual space position (latitude and longitude, or latitude, longitude and altitude) of a specified point (origin of the actual machine coordinate system) of the upper rotating body 42).

The real machine attitude sensor 416 outputs a signal corresponding to the real space attitude of the work machine 40 (e.g., the real space attitude in the real machine coordinate system) as well as the attitude of the working mechanism 44 in the real machine coordinate system. The attitude of the working mechanism 44 includes the hoisting angle of the boom 441 relative to the upper rotating body 42, the rotation angle of the arm 443 relative to the boom 441, and the rotation angle of the bucket 445 relative to the arm 443. If the position and attitude of the real machine coordinate system are defined fixedly relative to the lower traveling body 41 rather than the upper rotating body 42, the attitude of the working mechanism 44 may also include the rotation angle of the upper rotating body 42 relative to the lower traveling body 41.

As shown in FIG. 3, the working mechanism 44 includes a boom 441 that is movably attached to the upper rotating body 42, an arm 443 that is rotatably connected to the tip of the boom 441, and a bucket 445 that is rotatably connected to the tip of the arm 443. The working mechanism 44 is equipped with a boom cylinder 442, an arm cylinder 444, and a bucket cylinder 446 that are configured as extendable hydraulic cylinders.

The boom cylinder 442 is interposed between the boom 441 and the upper rotating body 42 so as to extend and retract when supplied with hydraulic oil, thereby rotating the boom 441 in the hoisting direction. The arm cylinder 444 is interposed between the arm 443 and the boom 441 so as to extend and retract when supplied with hydraulic oil, thereby rotating the arm 443 around a horizontal axis relative to the boom 441. The bucket cylinder 446 is interposed between the bucket 445 and the arm 443 so as to extend and retract when supplied with hydraulic oil, thereby rotating the bucket 445 around a horizontal axis relative to the arm 443.

(function)
The functions of the work support device and the imaging function control system having the above-mentioned configuration will be described with reference to the flowchart shown in Fig. 4. In the flowchart, a block "C" is used for the sake of simplicity, and means transmission and/or reception of data, and means a conditional branch in which processing in a branching direction is executed on the condition that the data is transmitted and/or received.

In the remote operation device 20, the remote control device 200 transmits a request to confirm the actual environment to the work support device 10 via the remote wireless communication device 224 (FIG. 4/STEP 210). The presence or absence of a designation operation via the remote input interface 210 is determined by the operator, and the request to confirm the actual environment may be transmitted if the determination result is positive. A "designation operation" is, for example, an operation such as tapping on the remote input interface 210 for the operator to designate the work machine 40 that he or she intends to remotely operate.

When a request to confirm the actual machine environment is received by the work support device 10, the request to confirm the actual machine environment is transmitted from the work support device 10 to the corresponding work machine 40 (Figure 4/C10).

When the work machine 40 receives a request to confirm the actual machine environment through the actual machine wireless communication device 424 (FIG. 4/C40), the actual machine control device 400 transmits captured image data representing the captured image (which may have been subjected to appropriate image processing) acquired through the actual machine imaging device 412 to the work support device 10 (FIG. 4/STEP 410). In addition, the actual machine control device 400 transmits actual machine position and orientation data representing the actual machine position and orientation to the work support device 10 (FIG. 4/STEP 412). The actual machine position and orientation include the real space position of the work machine 40 measured by the actual machine positioning device 414 and the real space orientation of the work machine 40 measured based on the output signal of the actual machine orientation sensor 416 and the orientation of the work mechanism 44.

When the work support device 10 receives the captured image data and the actual machine position and orientation data (FIG. 4/C12), the working mechanism space occupation mode recognition element 122 recognizes the spatial occupation mode of the working mechanism 44 based on the actual machine position and orientation data (FIG. 4/STEP 112). Specifically, the positions in the actual machine coordinate system of one or more designated points of the working mechanism 44 (e.g., the center point EP ₀ of the cutting edge of the bucket 445, both end points EP ₁ and EP ₂ of the cutting edge of the bucket 445, a point on the opening edge of the bucket 445 and/or a point on the base end of the bucket 445 (a point on the tip of the arm 443)) and/or the extension mode of a designated line segment (e.g., the cutting edge E (tip edge) of the bucket 445, a point on the opening edge of the bucket 445 and/or a point on the contour of the arm 443) are recognized as the spatial occupation mode of the working mechanism 44. The extension manner of each line segment and curved surface is expressed or specified by a relational expression of coordinate values (x, y, z) representing the line segment.
In addition, the spatial occupation mode of the work mechanism 44 may be recognized (measured/estimated) by the actual machine control device 400, and spatial occupation mode data representing the recognized spatial occupation mode of the work mechanism 44 may be transmitted from the work machine 40 to the work support device 10, so that the work mechanism spatial occupation mode recognition element 122 may recognize the spatial occupation mode of the work mechanism 44 based on the data (see FIG. 4 / STEP 112).

The work mechanism space occupancy recognition element 122 determines whether the cutting edge E of the bucket 445 is on the ground or underground (FIG. 4/STEP 120). For example, if at least one of the following conditions is satisfied: the height position (Z coordinate component) of the cutting edge E in the actual machine coordinate system is lower than the height position of the lower end of the lower travelling body 41, the height position of the cutting edge E in the work environment image coordinate system is lower than a predetermined height position, and the cutting edge E is not reflected in the work environment image, the cutting edge E is determined to be underground. In addition, the measurement result of the position of the ground surface (mainly the vertical position or height position) in the actual machine coordinate system by a distance measuring device (LiDAR and/or stereo camera, etc.) mounted or installed on the work machine 40 may be compared with the measurement result of the position of the cutting edge E of the bucket 445 (mainly the vertical position or height position) calculated from the attitude of the work mechanism 40 to determine whether the cutting edge E is underground.

In addition, if at least one of the following conditions is satisfied: unusual frequency components appear in the time series of the position of a designated point (e.g., the cutting edge center point EP ₀ ) of the work mechanism 44 in the real space coordinate system or the actual machine coordinate system; unusual frequency components appear in the sound around the bucket 445 collected by the microphone; and unusual frequency components appear in the sound in the time series of the strain amount at the designated point of the work mechanism 44 measured by the strain sensor, the cutting edge E may be determined to be underground.

When it is determined that the cutting edge E of the bucket 445 is on the ground (FIG. 4/STEP 120...1), the work support device 10 transmits work environment image data corresponding to the captured image to the remote control device 20 (FIG. 4/STEP 121). The work environment image data is not only the captured image data itself, but also image data representing a simulated work environment image generated based on the captured image.

When the remote operation device 20 receives work environment image data through the remote wireless communication device 224 (FIG. 11/C21), the remote control device 200 outputs an environmental image corresponding to the work environment image data to the remote image output device 221 (FIG. 4/STEP 221). As a result, for example, a work environment image showing the situation ahead of the work machine 40 (without superimposing the first and second index images) that includes the bucket 445 of the work mechanism 44 is output to the remote image output device 221 (see FIG. 6).

When it is determined that the cutting edge E of the bucket 445 is in the ground (FIG. 4/STEP 120..2), the working mechanism space occupancy mode recognition element 122 recognizes (predicts or estimates) the position of the cutting edge E of the bucket 445 when it appears on the ground surface (FIG. 4/STEP 122). For example, based on the time series of the hoisting angle of the boom 441 relative to the upper rotating body 42 over the immediately preceding designated past period, the time series of the rotation angle of the arm 443 relative to the boom 441, and the time series of the rotation angle of the bucket 445 relative to the arm 443, the position of the cutting edge E when it appears on the ground surface is recognized in a forward kinematic manner. FIG. 5 shows the cutting edge P ₁ (E) appearing on the ground surface, which is the extension mode of the cutting edge E on the ground surface P ₀ when the cutting edge E in the ground appears on the ground surface P ₀ .

In response to this, the work support image generating element 124 determines the superimposition mode of a first index image in the work environment image coordinate system (or captured image coordinate system) that represents the position or extension mode of the cutting edge E when it appears on the ground surface. Similarly, the work support image generating element 124 may determine the superimposition mode of a second index image in the work environment image coordinate system that represents the position or extension mode of the cutting edge E as a result of vertically projecting the cutting edge E onto the ground surface. Figure 5 shows the ground surface projected cutting edge _P2 (E) that is the result of projecting the cutting edge E underground in the vertical direction onto the ground surface _P0 .

To determine this overlap pattern, the extension pattern of the cutting edge appearing on the ground _P1 (E) (and the cutting edge projected on the ground _P2 (E)) in the real space coordinate system or real machine coordinate system is coordinate-converted into the extension pattern in the work environment image coordinate system. This coordinate conversion uses information on the position and orientation of the real machine coordinate system in the real space coordinate system, as well as the position and orientation of the real space coordinate system or real machine coordinate system of each point on the surface of an object, such as the ground surface, reflected in the work environment image measured by a distance sensor.

Then, a work support image is generated by the work support image generation element 124, and work support image data representing the work support image is transmitted to the remote control device 20 (FIG. 4/STEP 124). The work support image is an image resulting from superimposing the first index image (or the first index image and the second index image) on the work environment image.

When the remote operation device 20 receives work support image data via the remote wireless communication device 224 (Fig. 4/C22), the remote control device 200 outputs a work support image corresponding to the work support image data to the remote image output device 221 (Fig. 4/STEP 222).

As a result, for example, as shown in FIG. 6, a work support image in which the bucket 445 of the work mechanism 44 is reflected and the first index images _M1 ( _P1 ( _EP1 )) and _M1 ( _P1 ( _EP2 )) are superimposed on a work environment image showing the situation in front of the work machine 40 is output to the remote image output device 221. One first index image _M1 (P1 ( _EP1 )) shown in FIG. 6 is a three-dimensional figure having a substantially inverted cone shape whose apex position indicates the position of one end point of the cutting edge _P1 (E) (see FIG. 5) emerging on the ground. The other first index image _M1 ( _P1 ( _EP2 )) shown in FIG. 6 is a three-dimensional figure having a substantially inverted cone shape whose apex position indicates the position of one end point of the cutting edge _P1 (E) (see FIG. 5) _emerging on the ground.

As shown in Figure 7, a work support image in which the bucket 445 of the work mechanism 44 is reflected and a first index image _M1 ( _P1 (E)) is superimposed on a work environment image showing the situation ahead of the work machine 40 may be output to the remote image output device 221. The first index image _M1 ( _P1 (E)) shown in Figure 7 is a curved line segment figure that represents the extension pattern along the ground surface of the cutting edge _P1 (E) (see Figure 5) emerging on the ground surface.

As shown in Figure 8, a work support image in which the bucket 445 of the work mechanism 44 is reflected and the first index image _M1 ( _P1 ( _EP0 )) and the second index image _M2 ( _P2 ( _EP0 )) are superimposed on a work environment image showing the situation in front of the work machine 40 is output to the remote image output device 221. The first index image _M1 ( _P1 (EP0)) shown in Figure 8 is a three-dimensional figure having a substantially inverted cone shape whose apex position indicates the position of the center point of the cutting edge _P1 (E) (see Figure 5) emerging on the ground. The second index image _M2 ( _P2 ( _{EP0 )} ) shown in Figure 8 is a three-dimensional figure having a substantially inverted cone shape whose apex position indicates the position of the center point of the cutting edge _P2 (E) (see Figure 5) projected on the ground.

The shape and/or size of the first index image and the second index image may be changed in various ways. For example, as long as the first index image and/or the second index image functions as a pointer indicating the position of the target point, they may be various three-dimensional shapes such as a sphere, an elliptical sphere, a rectangular parallelepiped, a frustum, a torus, or a combination of these, or may be a line segment shape such as an arrow symbol.

As shown in Figure 9, a work support image in which the bucket 445 of the work mechanism 44 is reflected and a first index image _M1 (Tr(E)) is superimposed on a work environment image showing the situation ahead of the work machine 40 may be output to the remote image output device 221. The first index image _M1 (Tr(E)) shown in Figure 9 is a closed surface figure that represents the extension mode along the ground surface of an area whose front and back are defined by line segments along the ground surface corresponding to the cutting edge appearing on the ground _P1 (E) and the projected cutting edge _P2 (E) (see Figure ₅ ), respectively, and which is defined by another pair of line segments along the ground surface connecting the endpoints of the pair of line segments representing _the cutting edge appearing on the ground P1 (E) and the projected cutting edge P2 (E). This region corresponds to the projection onto the ground surface of a "continuous" or "quasi-continuous" time series of the spatial occupation pattern of the cutting edge E (the tip of the working mechanism 44) from underground to emerging at the ground surface.

The closed surface may be defined as a surface defined by multiple control points, such as a Bezier surface and/or a NURBS (Non-Uniform Rational B-Spline) surface. The surface may be defined as a surface having continuity (G1 continuity, G2 continuity, or G3 continuity). For example, when the closed surface is defined by a Bezier triangular surface, the domain of the control net of the Bezier triangular surface is defined by a triangular mesh stretched on a horizontal plane, and appropriate points are set as control points, and the closed surface is defined so that the continuity of the triangular patch is ensured.

A work support image may be generated by superimposing a first index image corresponding to the projection onto the ground surface of the "discrete" or "intermittent" time series of the spatial occupation mode of the cutting edge E (the tip of the working mechanism 44) from underground until it emerges from the ground to the ground surface, on the work environment image. In this case, a pair of line segments representing the cutting edge emerging on the ground surface _P1 (E) and the cutting edge projected on the ground surface _P2 (E), respectively, and one or more line segments sandwiched between the pair of line segments (corresponding to the vertical projection onto the ground surface _P0 of the intermittently displacing cutting edge E) are superimposed onto the work environment image as the first index image.

As shown in Figure 10, a work support image in which the bucket 445 of the work mechanism 44 is reflected and the first index images _M1 (Tr(E)) and _M1 (Tr(E)+) are superimposed on a work environment image showing the situation ahead of the work machine 40 may be output to the remote image output device 221. The first index image _M1 (Tr(E)+) shown by a dotted line in Figure 10 is a line drawing that is tangent to the closed surface (see Figure 9) as the first index image _M1 (Tr(E)) and extends along the ground surface in the operating direction (front-back direction) of the work mechanism 44.

The operator can control the operation of the work mechanism 44, and ultimately the bucket 445, by operating the operating levers that constitute the remote operation mechanism 211 while viewing the work support image output to the remote image output device 221. In the remote operation device 20, the remote control device 200 recognizes the operation mode of the remote operation mechanism 211, and a remote operation command corresponding to the operation mode is transmitted to the work support device 10 via the remote wireless communication device 224 (FIG. 4/STEP 214).

When the remote operation command is received by the work support device 10, the remote operation command is transmitted to the work machine 40 (Fig. 4/C14).

In the work machine 40, when an operation command is received by the actual machine control device 400 via the actual machine wireless communication device 422 (FIG. 4/C44), the operation of the work mechanism 44 and the like is controlled (FIG. 4/STEP 414). For example, the bucket 445 is used to dig up and scoop up soil and/or stones in front of the work machine 40, the upper rotating body 42 is rotated relative to the lower traveling body 41, and the soil is then dropped from the bucket 445 to an appropriate location.

(Action and Effect)
According to the work support device 10 exhibiting the above-mentioned functions, a work support image is generated in which the first index image and the second index image are superimposed on the work environment image (see Figs. 6 to 10). The "work environment image" is an image obtained through the actual machine imaging device 412, which represents the situation around the work machine 40, in particular, an image representing a situation in which the cutting edge E of the bucket 445 (the tip of the work mechanism 44) is hidden underground. The "first index image" is an image representing the projection result onto the ground surface of the spatial occupation state of the cutting edge E of the bucket 445 when it appears on the ground surface, which is predicted according to the operating state or operation state of the work mechanism 44 (see Fig. 5).

Thus, although the work mechanism 44 is hidden underground (or not reflected) in the work environment image, the projection result of the spatial occupation state of the blade tip of the bucket 445, which is the tip of the work mechanism 44, when it appears on the ground surface, which is predicted in consideration of the operating state of the work mechanism 44, can be recognized by an operator or the like who sees the work support image (see Figures 6 to 10).

The "second index image" is an image that represents or points to the projection onto the ground surface of the spatial occupation mode of the blade tip E of the bucket 445 that is hidden underground. This allows an operator or the like who comes into contact with the work support image to recognize the projection onto the ground surface of the spatial occupation mode of the blade tip E of the bucket 445, which is the tip of the work mechanism 44 that is hidden underground in the work environment image.

Other Embodiments of the Invention
In the above embodiment, the work support device 10 is configured as a computer separate from the remote control device 20 and the work machine 40. However, in other embodiments, the work support device 10 may be mounted on the remote control device 20 or the work machine 40.

Another embodiment in which the work support image generation function of the work support device 10 is performed by the remote operation device 20, and ultimately the remote control device 200, will be described using the flowchart shown in FIG. 11. In this flowchart, the block "C●" is used for simplification, as in FIG. 4, and means transmission and/or reception of data, and means a conditional branch in which processing in a branching direction is executed on the condition that the data is transmitted and/or received. For processing with the same content as in FIG. 4, the same numbers are assigned to "STEP●" and "C●", and the description will be simplified or omitted.

In the work machine 40, the actual machine control device 400 transmits captured image data representing the captured image acquired through the actual machine imaging device 412 to the work support device 10 (FIG. 11/STEP 410). When the captured image data is received by the work support device 10 (FIG. 4/C11), work environment image data corresponding to the captured image is transmitted from the work support device 10 to the remote control device 20 (FIG. 11/STEP 110). The work environment image data is not only the captured image data itself, but also image data representing a simulated work environment image generated based on the captured image.

When the remote operation device 20 receives work environment image data through the remote wireless communication device 224 (Fig. 11/C21), the remote control device 200 outputs an environmental image corresponding to the work environment image data to the remote image output device 221 (Fig. 11/STEP 212). As a result, a work environment image showing the situation ahead of the work machine 40 (with the first index image and second index image not yet superimposed) that includes, for example, the bucket 445 of the work mechanism 44 is output to the remote image output device 221 (see Figs. 6 to 10).

The actual machine position and orientation data representing the actual machine position and orientation is transmitted from the actual machine control device 400 to the work support device 10 (FIG. 11/STEP 412). When the actual machine position and orientation data is received by the work support device 10 (FIG. 11/C12), the work mechanism space occupation mode recognition element 122 recognizes the space occupation mode of the work mechanism 44 based on the actual machine position and orientation data (FIG. 11/STEP 112).

The work mechanism space occupancy recognition element 122 determines whether the cutting edge E of the bucket 445 is on the ground or underground (FIG. 11/STEP 120). If it is determined that the cutting edge E of the bucket 445 is on the ground (FIG. 11/STEP 120...1), the work environment image data is not transmitted from the work support device 10 to the remote control device 20.

On the other hand, if it is determined that the cutting edge E of the bucket 445 is underground (FIG. 11/STEP 120..2), the work mechanism space occupancy mode recognition element 122 recognizes (predicts or estimates) the position of the cutting edge E of the bucket 445 when it appears on the ground (FIG. 11/STEP 122). In response to this, the work support image generation element 124 determines the superimposition mode of the first index image (or the first index image and the second index image) in the work environment image coordinate system (or the captured image coordinate system).

Then, the work support device 10 transmits an "index image output instruction" indicating the manner in which the first index image and the second index image are to be superimposed on the work environment image to the remote control device 20 (FIG. 11/STEP 126).

When the remote operation device 20 receives an index image output instruction via the remote wireless communication device 224 (Fig. 11/C22), the remote control device 200 generates a work support image by superimposing the first index image (or the first index image and the second index image) on the work support image in accordance with the index image output instruction, and outputs the work support image to the remote image output device 221 (Fig. 11/STEP 222).

In the above embodiment, the work machine 40 is remotely operated by the operator via the remote control device 20, but in another embodiment, the work machine 40 may be actually operated by an operator on board the work machine 40. In this case, the work support device 10 transmits work support image data or an instruction to output an index image to the work machine 40 (see FIG. 4/STEP 122 or FIG. 11/STEP 122), and a work support image corresponding to the data may be output to the actual machine image output device constituting the actual machine output interface 420 (see FIGS. 7 to 11).

The work support image generated by the work support device 10 may be output to a communication terminal with a display device such as a smartphone or tablet that is carried by a worker and/or construction manager. If the communication terminal is equipped with an imaging device and is configured to be able to acquire the position information of the terminal using GNSS or the like, and to acquire the direction information and angle information of the terminal using a direction sensor and a tilt sensor or the like, it is possible to superimpose the work support image on the work environment image as in the above embodiment, allowing the spatial occupancy mode of the specified object to be understood.

10... Work support device 102... Database 122... Work mechanism space occupancy mode recognition element 124... Work support image generation element 20... Remote operation device 200... Remote control device 210... Remote input interface 211... Remote operation mechanism 220... Remote output interface 221... Remote image output device 222... Remote sound output device 224... Remote wireless communication device 40... Work machine 41... Lower traveling body 42... Upper rotating body 42C... Cab (operating room)
44: working mechanism 445: bucket 400: actual machine control device 410: actual machine input interface 412: actual machine imaging device 420: actual machine output interface M ₁ : first index image M ₂ : second index image.

Claims (9)

A work support system that generates a work support image in which a work environment image showing a situation in which a tip of a work mechanism is hidden underground, the image being acquired by an imaging device around a work machine having a work mechanism, and a first index image that shows the projection result onto the ground surface of the spatial occupancy mode of the tip of the work mechanism when the tip of the work mechanism emerges above the ground surface, which is predicted depending on the operation mode or operation mode of the work mechanism. The work support system according to claim 1,
A work support system that generates the work support image in which the first indicator image is superimposed on the work environment image as a line diagram that is the projection result of the contour or tip edge of the tip of the work mechanism onto the ground surface when the tip of the work mechanism is predicted to appear on the ground surface according to the operation mode or operation mode of the work mechanism, or as a pointer pointing to the projection result of one or more points that constitute the contour or tip edge of the tip of the work mechanism onto the ground surface when the tip of the work mechanism is predicted to appear on the ground surface according to the operation mode or operation mode of the work mechanism. The work support system according to claim 1,
A work support system that generates the work support image, which includes a second index image that represents or points to the projection result onto the ground surface of the space occupancy pattern of the tip part of the work mechanism hidden underground. The work support system according to claim 3,
A work support system that generates a work support image in which the second indicator image is superimposed on the work environment image as a line drawing that is a projection onto the ground surface of the contour or tip edge of the tip of the work mechanism that is hidden underground, or as a pointer that points to the projection onto the ground surface of one or more points that constitute the contour or tip edge of the tip of the work mechanism that is hidden underground. The work support system according to claim 1,
A work support system that generates the work support image in which the first index image, which represents the projection result onto the ground's surface of a time series of the spatial occupation mode of the tip of the work mechanism from underground until the tip of the work mechanism emerges from underground on the ground surface, predicted in accordance with the motion mode or operation mode of the work mechanism, is superimposed on the work environment image. The work support system according to claim 5,
A work support system that generates the work support image in which the first index image, which is a figure representing a closed surface resulting from the time-series projection of the tip edge of the tip of the work mechanism onto the ground surface, is superimposed on the work environment image. 7. The work support system according to claim 6,
A work support system that generates the work support image in which a line drawing that is tangent to the closed surface as the first index image and extends along the ground surface in the operating direction of the work mechanism is further superimposed on the work environment image. acquiring, via an imaging device, a work environment image around a work machine having a work mechanism, the image representing a situation in which a tip portion of the work mechanism is hidden underground;
and generating a work support image by superimposing a first index image, which represents the projection result onto the ground surface of the spatial occupation mode of the tip of the work mechanism when the tip of the work mechanism appears on the ground surface, which is predicted according to the motion mode or operation mode of the work mechanism, on the work environment image. A display device used when remotely operating a work machine having a work mechanism,
A display device displays a work support image in which a first index image representing the projection result onto the ground surface of the spatial occupation mode of the tip of the work mechanism when the tip of the work mechanism emerges above the ground surface, which is predicted based on the operation mode or operating mode of the work mechanism, is superimposed on a work environment image representing a situation in which the tip of the work mechanism is hidden underground, which is acquired through an imaging device around the work machine.

Images