JPH11213154A - Remote control assisting device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To remarkably improve controllability of remote control operation and to enhance work efficiency as well by objectively grasping perspective sensation from a camera mounted on a work machine as an object of remote control to an object of work and size of an object in an image. SOLUTION: A multilens camera 1 is installed at a work part 4 of the work machine A to be remotely controlled and two black-and-white images of the object 3 as the object of work are picked up with prescribed parallax by the multilens camera 1. Two picked up pictures are formed into a distant image based on the prescribed parallax by a distant image generating part 6 and colors according to the distance by every pixel are allocated to the distant image. The black-and-white image is superposed on the distance image by painting the colors allocated to the pixels corresponding to each pixel of one black-and-white image, a synthetic image as a pseudo three-dimensional image is generated and is transmitted to the side of a remote controller B by an image synthesizing part 7. The transmitted synthetic image is outputted and displayed on an image monitor 13 in the remote controller B so as to assist the remote control by a remote operator.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method for transmitting an image of a work object captured by a work machine to a remote control device wirelessly or by wire, and using the transmitted work image to remotely control the work machine. The present invention relates to a remote control support device that supports.

[0002]

2. Description of the Related Art Conventionally, when a work machine such as a backhoe working machine or a loading work machine is remotely controlled by a radio control or the like in a place where there is a risk, a person who remotely controls the work machine must be at least 50 meters from the work machine. When the user is away, the state of the work object such as excavated soil cannot be visually observed, so that the state of the work object cannot be grasped and remote control cannot be performed.

For this reason, a camera for capturing the state of a work target is mounted on a work machine, and the captured image is wirelessly transmitted to a remote control device for performing remote control, and a remote operator uses the image based on the image. By indirectly grasping the state of the work target, the work can be performed even if the remote operator is more than 50 meters away from the work machine.

[0004]

However, since the image transmitted to the remote control device is a two-dimensional image, the sense of distance from the camera mounted on the work machine to the work target and the size of the object in the image are large. Therefore, there is a problem that the operability and efficiency of remote work are extremely reduced, and a desired work cannot be performed reliably.

[0005] In particular, there is a problem in that the work target of the working machine is unevenness of soil or the like, so that it is difficult to grasp the distance, and an accident due to remote control is likely to occur.

Accordingly, the present invention has been made to solve the above problem and to objectively grasp the sense of distance from a camera mounted on a work machine to be remotely controlled to a work target and the size of an object in an image. It is an object of the present invention to provide a remote control support device capable of significantly improving the controllability of the remote control operation and improving the operation efficiency.

[0007]

A first aspect of the present invention is a remote control support for remotely controlling a work machine based on an image of a work target transmitted from the remotely controlled work machine to a remote control device. In the apparatus, the multi-view camera mounted on the work machine and imaging the work target as a plurality of black-and-white images having a predetermined parallax, and the predetermined parallax from the plurality of black-and-white images imaged by the multi-view camera are also provided. A distance image generating means for obtaining a distance between the multi-lens camera and the work object for each pixel constituting the black and white image, and generating a distance image having color information assigned according to the distance; ,
Composite image generating means for generating a composite image in which the color information of the distance image is superimposed on each pixel on the monochrome image, and composite image output means mounted on the remote control device and displaying and outputting the composite image; It is characterized by having.

Accordingly, the remote operator can remotely control the vehicle based on the synthesized image having the distance information, and can easily grasp the objective feeling of distance to the work target and the size of the object. The operability of remote work can be greatly improved.

In a second aspect based on the first aspect, the work machine has the distance image generating means and the composite image generating means, and the remote control device transmits the composite image generated by the composite image generating means. The remote control device further includes a receiving unit that receives the composite image transmitted by the transmitting unit.

[0010] This eliminates the need for the remote control device to generate a distance image and / or a composite image.
Since only the composite image needs to be displayed and output, the remote control device can be reduced in size, and the portability can be improved.

[0011]

Embodiments of the present invention will be described below with reference to the drawings.

FIG. 1 is a diagram showing a schematic configuration of a remote control system including a remote control support device according to an embodiment of the present invention. This remote control system mainly includes a work machine A that is remotely controlled and a remote control device B that performs remote control.

In FIG. 1, a working machine A includes a working section 4 and a working machine main body 5. The working part 4 is a part for performing work such as excavation. In FIG. 1, a boom 4c attached to the working machine main body 5 is driven by a boom cylinder 4d, and an arm 4b is supported at a tip of the boom 4c. This is a backhoe working machine having a bucket 4a supported at the tip of the arm 4b. The working unit 4 has a multi-lens camera 1 that captures an image of a subject 3 to be worked by the working unit 4, and is attached to an arm 4b in FIG. The mounting position of the multi-lens camera 1 may be set at a position where the subject 3 falls within the predetermined angle of view 2.

The multi-lens camera 1 has a plurality of cameras arranged at a predetermined interval in one multi-lens camera 1 and simultaneously captures the same subject.
This is realized by an OS type solid-state imaging device.

This multi-lens camera 1 is used when the number of cameras is two, as shown in FIGS. 2 (a) and 2 (b).
This is effective when obtaining the distance between a certain image 31 of one of the images 30a and 30b captured by the multi-lens camera 1, and a window 32 around the image 31.
The pattern most similar to the luminance value (brightness) pattern of the pixel 31 (including the pixel 31) is searched for on the excipolar line in the other image 30b, and the corresponding pattern 32 'is obtained. The position of a pixel in the image 30b can be obtained, and the distance can be obtained from the displacement of the pixel, that is, the parallax. The distance is set as the distance of the pixel 31, and the other pixels similarly have information on the distance, so that An image having distance information corresponding to the image 30a can be obtained.

A black-and-white image (hereinafter, referred to as a black-and-white image) picked up by the multi-lens camera 1 is input to a distance image generating section 6 and an image synthesizing section 7 of the working machine body 5 via a line 1a.

As described above, the distance image generation unit 6 generates a distance image by using predetermined parallaxes of monochrome images corresponding to the number of cameras captured by the multi-lens camera 1 (see FIG. 2) (FIG. 3). (B)).

FIG. 3A shows an example of one black and white image of the number of cameras captured by the multi-lens camera 1, and FIG. 3B shows the black and white image of FIG. The range image corresponding to the image is shown. The subject shown in FIG.
It is shown as an example to show the relationship between the black and white image and the distance image instead of the work target of the work machine A. The distance image shown in FIG. 3B divides a human subject 40 and its background 41 shown in FIG. 3A by a predetermined distance, for example, 2 meters, and sets a pixel having a distance of 2 meters or less as an area 50. ,
Pixels at a distance exceeding 2 meters are shown as a region 51.

The range image shown in FIG. 3B shows a case where the range resolution is 2, but the range resolution can be further increased. For example, assuming that the distance resolution is 100, as the distance becomes farther from the vicinity of the multi-lens camera 1, an appropriate color is assigned to each distance area, and the color assigned to each distance area is colored. An image is generated. This coloring is, as the distance increases from near to far, for example, if it is colored stepwise as red →... → yellow →... → blue, the area of the short distance portion is colored red like a signal, It suggests that it is close to the work target or a danger, etc., and makes it easy to grasp the distance intuitively.

On the other hand, the image synthesizing unit 7 includes a distance image generating unit 6
Thus, the distance image in which each distance region is colored and the black-and-white image transmitted from the multi-lens camera 1 are combined. That is, the colored color information of the distance image is superimposed on the monochrome image. In the composite image synthesized in this way, a color corresponding to the distance is colored with respect to the black-and-white image, and a more objective distance sensation is added to the distance sensation of the black-and-white image itself. 3D image.

The synthesized image generated by the image synthesizing section 7 is transmitted as radio information from the image transmission control section 8 via the antenna 11a and transmitted to the remote control device B side.

The work machine body 5 further includes a work machine control unit 9 for controlling each unit in the work machine A and a transmission / reception control unit 10. The work machine control unit 9 receives data wirelessly transmitted from the remote control device B side, which is input to the transmission / reception control unit 10 via the antenna 11c, and controls each unit in the work machine, that is, the drive control and traveling unit of the work unit 4. In addition to performing the drive control of the control unit 16, the control result or control state is transmitted as wireless data to the remote control device B via the transmission / reception control unit 10 and the antenna 11 c.

On the other hand, the remote control device B is a device for remote control of the remote operator, and includes an image reception control unit 12 for receiving a composite image wirelessly transmitted from the work implement A via the antenna 11b. It has an image monitor 13 for displaying and outputting an image, a remote operation input / output unit 14 for inputting an instruction of a remote operation operation of the work unit A or outputting a control result, and data relating to a remote operation of the work unit A via an antenna 11d. Transmission / reception control unit 1 for transmitting or receiving
5

As described above, the remote control system comprises the multi-lens camera 1, the range image generator 6, the image synthesizer 7, the image transmission controller 8, the antennas 11a and 11b, and the image reception controller 1.
2, an image transmission system including an image monitor 13, a work control unit 9, a transmission / reception control unit 10, antennas 11c and 11d,
A data transmission system including a transmission / reception control unit 15 and a remote operation input / output unit 14;
By referring to the composite image displayed and output in 3, it is possible to perform an accurate remote control operation with good operability using the remote control input / output unit 14 while grasping an objective sense of distance.

Next, a modification of the embodiment of the present invention will be described. FIG. 4 is a diagram showing the configuration of a first modification of the remote control system shown in FIG. The remote control system shown in FIG. 4 has the same configuration as the image synthesizing unit 7.
A differs from the remote control system shown in FIG. 1 in that A is arranged on the remote control device B side, and the other configuration is the same as that of the remote control system shown in FIG.

That is, in the remote control system shown in FIG. 4, on the work implement A side, the black-and-white image captured by the multi-lens camera 1 and the distance image generated by the distance image generation unit 6 based on the black-and-white image are compared. The image data is transmitted to the remote control device B via the image transmission control unit 8 as it is.
The image combining unit 7A combines the transmitted monochrome image and the distance image, and the combined image is displayed and output from the image monitor 13.

Even with such a configuration, the same operation and effects as those of the remote control system shown in FIG. 1 are obtained.

FIG. 5 is a diagram showing the configuration of a second modification of the remote control system shown in FIG. 1. This remote control system has the same structure as the image synthesizer 7.
B and the distance image generation unit 6 having the same configuration as the distance image generation unit 6
1 is different from the remote control system shown in FIG. 1 in that B is arranged on the remote control device B side, and the other configuration is the same as that of the remote control system shown in FIG.

That is, in the remote control system shown in FIG. 5, on the work machine A side, the monochrome images corresponding to the number of cameras captured by the multi-lens camera 1 are directly transmitted to the remote control device B via the image transmission control unit 8. At the remote control device B side, the distance image generation unit 6B generates a distance image from the transmitted and received monochrome images of the number of cameras, and the image synthesis unit 7B synthesizes the distance image and the monochrome image. Produces
The generated composite image is displayed and output from the image monitor 13.

Even with such a configuration, the same operation and effect as those of the remote control system shown in FIG. 1 can be obtained.

In particular, in the first and second modifications, since the configuration of the image transmission system to be disposed on the work machine A side is small, the existing remote control work machine A can be easily applied to the present invention. Can be improved.

Conversely, in the remote control system shown in FIG.
Since the range image generating unit 6 and the image synthesizing unit 7 are mounted on the work machine A, the configuration of the remote control device B can be simplified, the size and weight can be reduced, and the portability can be improved.

In the above-described embodiment of the present invention, the transmission medium between the work implement A and the remote control device B is wireless, but the transmission medium is not limited to this, and may be wired. When this wire is applied, for example, in a place where wireless communication is difficult, and in a place where there is a danger, for example, a case where toxic gas is generated and a work in a sinus with a complicated shape is performed. .

In the embodiment of the present invention described above, the case where one multi-lens camera 1 is installed in the work machine A has been described. However, the present invention is not limited to this, and a plurality of multi-lens cameras may be installed. Good. For example, the state around work machine A when work machine A itself is moved by traveling unit 16 or when work unit 4 turns may be imaged.
In this case, a plurality of image monitors 13 may be provided, or a plurality of composite images may be displayed in a window.

[Brief description of the drawings]

FIG. 1 is a diagram showing a schematic configuration of a remote control system including a remote control support device according to an embodiment of the present invention.

FIG. 2 is a diagram illustrating a principle of generating a distance image using a multi-lens camera.

FIG. 3 is a diagram illustrating a specific example of a black-and-white image and a distance image.

FIG. 4 is a diagram showing a configuration of a first modification of the remote control system shown in FIG. 1;

FIG. 5 is a diagram showing a configuration of a second modification of the remote control system shown in FIG. 1;

[Explanation of symbols]

DESCRIPTION OF SYMBOLS 1 ... Multi-view camera 1a ... Line 2 ... Angle of view 3 ... Subject 4
… Working part 4a… Bucket 4b… Arm 4c… Boom 4d…
Boom cylinder 5 Work machine body 6 Distance image generation unit 7 Image synthesis unit 8 Image transmission control unit 9 Work machine control unit 10, 15
Transmission / reception control units 11a to 11d: Antenna 12: Image reception control unit 1
3. Image monitor 14 Remote input / output unit 16 Running unit

──────────────────────────────────────────────────の Continued on the front page (51) Int.Cl. ⁶ Identification code FI G09G 5/36 510 G06F 15/66 450

Claims (2)

[Claims] 1. A remote control support device for remotely controlling a work machine based on an image of a work object transmitted from a remotely controlled work machine to a remote control device, wherein the remote control support device is mounted on the work machine and the work A multi-view camera that captures an object as a plurality of black-and-white images having a predetermined parallax, and a plurality of black-and-white images captured by the multi-view camera, for each pixel constituting the black-and-white image based on the predetermined parallax. Find the distance between the multi-lens camera and the work object,
A distance image generating means for generating a distance image having color information assigned according to the distance; and a composite image generation for generating a composite image in which the color information of the distance image is superimposed for each pixel on the monochrome image. And a composite image output means mounted on the remote control device and displaying and outputting the composite image. 2. The work machine includes the distance image generation unit and the composite image generation unit, and further includes a transmission unit that transmits a composite image generated by the composite image generation unit to the remote control device. The remote control support device according to claim 1, wherein the remote control device further includes a receiving unit that receives the composite image transmitted by the transmitting unit.