JP7330926B2 - Filming system and remote control system - Google Patents

Description

The present invention relates to an imaging system and a remote control system.

In disaster restoration work due to earthquakes, floods, landslides, etc., work with manned machines may cause secondary disasters, so there are cases where unmanned machines that are remotely controlled are used. For example, in Patent Document 1, images of two fisheye cameras installed on a construction machine are displayed on a head-mounted display (HMD) worn by an operator in a remote control room with binocular parallax reproduced. A remote control system is described.

According to the remote control system described in Patent Document 1, since there is binocular parallax, it is possible to obtain a sense of depth and perspective that cannot be obtained with a monocular camera. In addition, since the fisheye lens is used, the field of view is wide, and the direction of the head-mounted display and the direction of the image are synchronized, so by shaking the head wearing the head-mounted display left and right, the desired direction can be changed. You can see the video of

JP 2018-121195 A

Here, there is a trade-off relationship between the image quality and the angle of view of the camera. A fisheye lens can capture a wide range of images, but there is a problem that the resolution is low because the sensor pixels of the camera cannot be used to the maximum. Therefore, it is sometimes difficult to ensure proper visibility when performing detailed work by remote control.

From such a point of view, the present invention provides a photographing system and a remote control system that can secure the visibility necessary for detailed work while having a wide field of view.

A photographing system according to the present invention is a photographing system for photographing images for remotely operating a work machine. This imaging system includes a stereo camera provided in the driver's cab of the working machine, and a first remote control device for remotely controlling the stereo camera.
The stereo camera has a fisheye lens and an optical zoom mechanism, and the optical zoom mechanism operates according to the operation of the first remote control device.

In the imaging system according to the present invention, since the stereo camera has a fisheye lens, it is possible to shoot in a super wide-angle range (for example, 180°). In addition, since the optical zoom mechanism can be controlled by remote control, it is possible to switch between wide-angle shooting and telephoto shooting. Therefore, it is possible to secure the visibility necessary for detailed work while having a wide field of view, making it easier to remotely operate the work machine than before.

The first remote control device has two foot pedals, the first foot pedal for zooming in and zoom out, and the second foot pedal for adjusting the zoom speed.

In this way, the operator can switch between wide-angle photography and telephoto photography and change the switching speed with his/her legs while operating the work machine by hand. Therefore, remote control of the work machine is easier.

Further, a remote control system according to the present invention includes the above-described imaging system, a head-mounted display for displaying an image captured by the imaging system, and a second remote control device for remotely controlling the working machine. It is.

In the imaging system according to the present invention, the operator can see an image (3D image) with a sense of depth that reproduces binocular parallax. In addition, since the optical zoom mechanism can be controlled by remote control, it is possible to switch between wide-angle shooting and telephoto shooting. Therefore, it is possible to secure the visibility necessary for detailed work while having a wide field of view, making it easier to remotely operate the work machine than before.

According to the present invention, it is possible to ensure visibility necessary for detailed work while having a wide field of view.

1 is a schematic configuration diagram of a remote control system according to an embodiment of the present invention; FIG. 1A and 1B are schematic configuration diagrams of an optical zoom mechanism, in which (a) shows a state during wide-angle photography, and (b) shows a state during telephoto photography; 1A and 1B are schematic configuration diagrams of a fish-eye optical zoom mechanism, in which (a) shows a state during wide-angle photography, and (b) shows a state during telephoto photography; FIG. 4 is a diagram showing an incident route of an image from a subject in a fisheye lens;

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments for carrying out the present invention will be described in detail with appropriate reference to the drawings. Each drawing is merely a schematic representation to the extent that the present invention can be fully understood. Accordingly, the present invention is not limited to the illustrated examples only. In addition, in each figure, the same code|symbol is attached|subjected about a common component and a similar component, and those overlapping description is abbreviate|omitted.

<Configuration of remote control system according to embodiment>
A configuration of a remote control system 1 according to an embodiment will be described with reference to FIG. FIG. 1 is a schematic configuration diagram of a remote control system 1. As shown in FIG. A remote control system 1 is a system for remotely controlling a work machine. The remote control system 1 can be used in various situations using working machines, and the working machine to be remotely controlled and the work performed using the working machine are not particularly limited.

As shown in FIG. 1, the remote control system 1 is mainly composed of devices mounted on the working machine and devices provided in the operation room. The working machine may be, for example, one that can be operated by an operator during normal times. The working machines are, for example, backhoes, vibration rollers, breakers, bulldozers, dump trucks, etc., which are types of construction machines.

The construction of the working machine includes a stereo camera 2 composed of a plurality of (here, two) cameras 20, 20, converters 3, 3 connected to each camera 20, and a communication device 4. .
The configuration of the control room includes a communication device 5, converters 6, 6, an information processing device 7, a remote control device 8 for remotely controlling the stereo camera 2, and a head mount for displaying the image of the stereo camera 2. A display 9 and a remote control device 10 for remotely controlling the work machine are provided. Since the transducers 6 are paired with the transducers 3, the number of transducers 6 is the same as that of the transducers 3 (here, two).

An operator operates the work machine by operating the remote control device 10 while wearing the head mounted display 9 on the head. Hereinafter, the remote control system 1 excluding the components for operating the work machine, that is, the components for capturing images for remotely operating the work machine will be collectively referred to as the "shooting system 1A". The imaging system 1A includes a stereo camera 2 and a remote control device 8. As shown in FIG.

The stereo camera 2 is installed, for example, in the driver's cab of the working machine, and captures images used for remote control of the working machine. The two cameras 20, 20 provided in the stereo camera 2 correspond to the left and right eyes of the operator who gets on the vehicle in normal times. The camera 20 preferably has a resolution that enables recognition of the situation at the work site, and a frame rate that does not interfere with remote control. Camera 20 may be, for example, a 4K resolution or higher video camera. Each camera 20 constituting the stereo camera 2 has an optical zoom mechanism 21 and a fisheye lens 22 . Optical zoom is a method of obtaining telephoto capability by optically changing the focal length by moving the lens. The optical zoom mechanism 21 has several types of lenses and can adjust the distance between the lenses. The fisheye lens 22 is a lens that enables shooting in a super-wide-angle range (for example, 180°).

Although various types of optical zoom mechanisms have been developed in the past, the remote control system 1 can use any type without particular limitation. It is desirable that the optical zoom mechanism 21 causes little image disturbance due to vibration or the like during movement of the work machine or during work. The optical zoom mechanism 21 preferably has a camera shake correction function.

The stereo camera 2 is connected to a communication device 4 via a LAN (Local Area Network) and receives control signals transmitted from the operation room. The stereo camera 2 is also connected to the converter 3 via an HDMI (registered trademark) (High-Definition Multimedia Interface) line, and transmits captured video to the converter 3 . Note that the HDMI line is an example, and the video line is not limited to the HDMI line. The video line may be a coaxial line, an SDI (Serial Digital Interface) line, or the like. The same applies to the connection between the converter 6 and the information processing device 7 . In that case, the converters 3 and 6 will convert to a format corresponding to the video line.

The converter 3 converts the video signal in HDMI format into IP (Internet Protocol) format. The converter 3 is connected to the communication device 4 via a LAN (Local Area Network), and transmits the converted video signal to the communication device 4 . A device (converter 6) similar to the converter 3 is also provided in the operation room, and the processing opposite to that of the converter 3 is performed.

The communication devices 4 and 5 are devices that have antennas and implement wireless communication. Communication device 4 is connected to stereo camera 2 and converter 3 as described above. The communication device 5 is connected to the information processing device 7 via a LAN, and transmits the control signal received from the information processing device 7 to the work machine by wireless communication. The communication device 5 is also connected to the converter 6 via a LAN, and transmits to the converter 6 a video signal received by wireless communication.

The converter 6 converts the IP format video signal into the HDMI format. The converter 6 is connected to the information processing device 7 via an HDMI line, and transmits the converted video signal to the information processing device 7 .

The information processing device 7 is a device that centrally controls the remote control system 1 . The information processing device 7 has, for example, a function related to image display for displaying images captured by the stereo camera 2 on the head mounted display 9 and a function related to control of the work machine and the stereo camera 2 . These functions provided in the information processing device 7 are realized by, for example, a CPU (Central Processing Unit) reading and executing a predetermined program from a ROM (Read Only Memory) (not shown) or the like. The information processing device 7 is connected to a remote controller 8, a head-mounted display 9, and a remote controller 10 via, for example, a USB (Universal Serial Bus) or DisplayPort.

A remote control device 8 is a device for remotely controlling the stereo camera 2 . The remote control device 8 in this embodiment includes two foot pedals 30A and 30B. The stereo camera 2 zooms in by depressing one foot pedal 30A. Also, by depressing the other foot pedal 30B, the stereo camera 2 zooms out. Note that the remote control device 8 may include components other than the foot pedals 30A and 30B, and may instruct the start and end of photographing by the stereo camera 2, for example. Foot pedals 30A and 30B transmit zoom-in/zoom-out commands to stereo camera 2 via information processing device 7 .

A head-mounted display 9 displays an image captured by the stereo camera 2 . A head-mounted display 9 is worn by an operator in the operation room. Images from the two cameras 20 and 20 constituting the stereo camera 2 are projected onto the head mounted display 9 with binocular parallax reproduced on the left eye portion and the right eye portion of the head mounted display 9 . In other words, the stereo camera 2 captures images of the object from different angles using the two fisheye lenses 22, 22, and the operator can see an image (3D image) with a sense of depth that reproduces binocular parallax. .

The head-mounted display 9 includes an angle sensor, a gyro sensor, and the like (not shown), and senses the direction, movement, etc. of the operator's head by the angle sensor, the gyro sensor, and the like. The sensed orientation and movement of the operator's head are used in conjunction with the image projected on the head-mounted display 9 . The type and number of sensors included in the head-mounted display 9 are not particularly limited as long as the operator wearing the head-mounted display 30 and the image to be projected can be linked.

As described above, according to the remote control system 1 according to the present embodiment, the field of view from the cab of the work machine can be faithfully projected on the head-mounted display 9 almost in real time. Since the operator can work while watching the image with a sense of depth almost in real time, the work machine can be operated accurately. In addition, since the stereo camera 2 constantly captures images of various meters installed in the driver's cab, the operator can obtain the same information as in the actual driver's cab by viewing the meters when necessary. .

A remote control device 10 is a device for remotely controlling a work machine. Although FIG. 1 exemplifies a joystick controller as the remote control device 10, the remote control device 10 is not limited to this.

<About the optical zoom mechanism>
The optical zoom mechanism 21 will be described with reference to FIG. 2A and 2B are schematic configuration diagrams of the optical zoom mechanism 21. FIG. 2A shows the state during wide-angle photography, and FIG. 2B shows the state during telephoto photography. FIG. 2 is an example of the optical zoom mechanism 21, and is a schematic diagram of a "two-group type". The optical zoom mechanism 21 shown in FIG. 2 exemplifies a model in which the convex lens 21a and the concave lens 21b are combined to simplify the explanation, but the convex lens 21a and the concave lens 21b may be configured as a lens group. The concave lens 21b (or lens group) plays a role of changing the focal length and is called a "variator". The convex lens 21a (or lens group) plays a role of adjusting the focal position and is called a "compensation center".

The angle of view of the camera 20 (wide angle or telephoto) depends on the focal length. In FIG. 2, the imaging range line is drawn with a thick solid line, and the focal pattern line is drawn with a thin dashed-dotted line.
During wide-angle photography, as shown in FIG. 2A, the convex lens 21a is moved closer to the image sensor 23 to shorten the focal length. At this time, the distance between the lenses is increased by separating the concave lens 21b from the convex lens 21a.
On the other hand, during telephoto shooting, as shown in FIG. 2B, the convex lens 21a is moved away from the image sensor 23 to increase the focal length. At this time, the distance between the lenses is shortened by bringing the concave lens 21b closer to the convex lens 21a.

<Construction in which a fisheye lens is combined with an optical zoom mechanism>
A configuration in which a fish-eye lens 22 is combined with an optical zoom mechanism 21 (referred to as a "fish-eye optical zoom mechanism") will be described with reference to FIG. 3A and 3B are schematic configuration diagrams of a fish-eye optical zoom mechanism, in which (a) shows the state during wide-angle photography, and (b) shows the state during telephoto photography. In FIG. 3, the imaging range line is drawn with a thick solid line, and the focal pattern line is drawn with a thin one-dot chain line.

During wide-angle photography, the imaging range line is positioned at the peripheral edge of the fisheye lens 22, as shown in FIG. 3(a). Here, as shown in FIG. 4, the fish-eye lens 22 can acquire an image from a subject within a range of about 180 degrees at maximum. Therefore, an object in a wide range is projected onto the image sensor 23, and ultra-wide-angle (eg, 180° angle of view) imaging is possible.
On the other hand, during telephoto shooting, the imaging range line is located near the center of the fisheye lens 22, as shown in FIG. 3(b). Therefore, an object in a narrow range (in front of the camera 20) is projected onto the image sensor 23, and the resolution of the object in this range is improved, enabling telephoto shooting.

As described above, in the photographing system 1A according to the present embodiment and the remote control system 1 using the photographing system 1A, the stereo camera 2 includes the fisheye lens 22. Therefore, photographing in a super wide-angle range (for example, 180°) is possible. is possible. Further, since the optical zoom mechanism 21 can be controlled by remote control, it is possible to switch between wide-angle shooting and telephoto shooting. Therefore, it is possible to secure the visibility necessary for detailed work while having a wide field of view, making it easier to remotely operate the work machine than before.

In particular, the remote control device 8 of this embodiment has two foot pedals 30A and 30B, and zooms in when one foot pedal 30A is depressed and zooms out when the other foot pedal 30B is depressed. . Since the operator can switch between wide-angle photography and telephoto photography with his leg while operating the work machine by hand, remote control of the work machine is easier.

Although the embodiment of the present invention has been described above, the present invention is not limited to this, and can be implemented within the scope of the claims.

The remote control device 8 of the embodiment zooms in by stepping on one foot pedal 30A, and zooms out by stepping on the other foot pedal 30B. However, the configuration of the remote control device 8 is not limited to this. For example, one foot pedal 30A may be used to operate zoom-in and zoom-out, and the other foot pedal 30B may be used to adjust the zoom speed.

1 remote control system 1A photography system 2 stereo camera 20 camera 21 optical zoom mechanism 22 fisheye lens 8 remote control device (first remote control device)
9 head mounted display 10 remote control device (second remote control device)
30A, 30B foot pedal

Claims (2)

A shooting system for shooting a video for remotely controlling a working machine,
a stereo camera provided in a cab of the working machine;
a first remote control device for remotely controlling the stereo camera,
The stereo camera has a fisheye lens and an optical zoom mechanism, and the optical zoom mechanism operates according to the operation of the first remote control device,
The first remote control device has two foot pedals,
a first foot pedal to operate zoom in and zoom out, and a second foot pedal to adjust the zoom speed;
An imaging system characterized by: The imaging system according to claim 1 ;
a head-mounted display that displays an image captured by the imaging system;
a second remote control device for remotely controlling the work machine;
A remote control system characterized by:

Images