JP2021022802A - Industrial vehicle remote operation system - Google Patents

Abstract

To provide an industrial vehicle remote operation system that even when a target of an operator departs from a camera picture in remote operation, inhibits sight of the target from being lost.SOLUTION: An industrial vehicle remote operation system comprises: an industrial vehicle comprising a camera; a remote operation device for remotely controlling the industrial vehicle; and a display unit 38 comprising a display screen for enabling a camera picture imaged by the camera to be displayed. The industrial vehicle remote operation system comprises: a target designation unit for designating a target displayed in the camera picture P1; a world coordinate acquisition unit for acquiring world coordinates of the target designated by the target designation unit; a world coordinate change unit for following movement of the industrial vehicle to change the world coordinates of the target; and a guide line drawing unit that on the camera picture P1, draws a guide line L between the world coordinates (Xwc, Ywc, Zwc) of the industrial vehicle and the world coordinates (Xwt, Ywt, Zwt) of the target changed by the world coordinate change unit.SELECTED DRAWING: Figure 5

Description

The present invention relates to a remote control system for an industrial vehicle.

As a conventional technique related to a remote control system for an industrial vehicle, for example, a remote control system for a wirelessly operated vehicle disclosed in Patent Document 1 is known. The wirelessly operated vehicle remote control system disclosed in Patent Document 1 is a system in which an operator remotely controls a wirelessly operated vehicle located away from the base station by an operating device from the base station. The operation device includes a hub connected to a LAN cable, a base-side control unit connected to the hub, an operation pedal connected to the base-side control unit, an operation handle, a control box, a display monitor, a speaker, and the like.

The display monitor displays an image of the surroundings of the vehicle taken by a camera mounted on the wirelessly operated vehicle, information detected by various sensors described later, and the like. In Patent Document 1, one of the two display monitors is a display monitor that displays an image taken by a camera, and the other is a display monitor that displays vehicle information detected by various sensors. The wirelessly operated vehicle is equipped with a vehicle control device and an in-vehicle wireless device, as well as a sensor device equipped with a camera, various sensors, and the like, and wireless operations such as traveling and tilting of the vehicle. It is equipped with actuators that operate various devices related to the operation of the type vehicle.

Japanese Unexamined Patent Publication No. 2014-11518

However, in the remote control system of the wirelessly operated vehicle disclosed in Patent Document 1, if the operator's target deviates from the camera image displayed on the display monitor due to the running of the wirelessly operated vehicle, the operator loses sight of the target. There is a problem.

The present invention has been made in view of the above problems, and an object of the present invention is to provide a remote control system for an industrial vehicle in which the target is not lost even if the operator's target deviates from the camera image in remote control. is there.

In order to solve the above problems, the present invention includes an industrial vehicle provided with a camera, a remote control device for remotely operating the industrial vehicle, and a display screen capable of displaying a camera image taken by the camera. In a remote control system of an industrial vehicle equipped with a display monitor, a target designation unit that specifies a target displayed in the camera image and a world coordinate that acquires the world coordinates of the target specified by the target designation unit. The acquisition unit, the world coordinate changing unit that changes the world coordinates of the target according to the movement of the industrial vehicle, and the world coordinates of the industrial vehicle and the world coordinates of the target changed by the world coordinate changing unit . It is characterized by having a guideline drawing unit for drawing the guideline between them on the camera image.

In the present invention, a guideline is drawn between the world coordinates of the target designated by the target designation unit and the world coordinates of the industrial vehicle in the camera image. Even if the target deviates from the camera image due to the running of the industrial vehicle, the world coordinates of the target are changed according to the movement of the industrial vehicle by the world coordinate change part, and the guideline in the camera image is the existence of the target deviating from the camera image. Indicates the direction to do. Therefore, since the operator who performs the remote control can recognize the direction in which the target exists, the target is not lost even if the target deviates from the camera image.

Further, in the remote control system of the industrial vehicle, the target designation unit may be a touch panel capable of touch operation on the display screen.
In this case, the operator can easily specify the target displayed on the camera image by touching the touch panel.

Further, in the remote control system of the industrial vehicle, the guideline may be drawn by a straight line in the camera image.
In this case, since the guideline is a straight line, the operator can intuitively recognize the direction in which the target exists.

According to the present invention, it is possible to provide a remote control system for an industrial vehicle in which an operator's target does not lose sight of the target even if the target deviates from the camera image in remote control.

It is a block diagram which shows the electrical structure of the remote control system of the forklift which concerns on embodiment of this invention. It is a side view of the forklift which concerns on embodiment of this invention. It is a perspective view of the forklift which concerns on embodiment of this invention. It is explanatory drawing explaining the world coordinates of a forklift. It is a figure of the camera image displayed on the display part of a remote control system. It is a figure of the camera image displayed on the display part of a remote control system. This is an example of a guideline for a modified example.

Hereinafter, the remote control system for an industrial vehicle according to the embodiment of the present invention will be described with reference to the drawings. This embodiment is an example of a remote control device for a forklift as an industrial vehicle.

As shown in FIG. 1, the forklift remote control system 10 includes a reach-type forklift (hereinafter, simply referred to as “forklift”) 11 and a remote control device 12 used for remotely controlling the forklift 11. ing. In this embodiment, the forklift 11 is arranged in the work place E. The remote control device 12 is arranged in an operation room R provided separately from the work place E. Then, the operator can remotely control the forklift 11 of the work place E from the operation room R by using the remote control device 12.

As shown in FIG. 2, the vehicle body 13 of the forklift 11 is provided with a pair of left and right reach legs 14. The pair of left and right reach legs 14 extend from the front portion of the vehicle body 13 toward the front. Front wheels 15 are provided on the front portions of the pair of left and right reach legs 14. Rear wheels 16 and caster wheels (not shown) are provided at the rear of the vehicle body 13. The rear wheel 16 is a steerable drive wheel.

The forklift 11 includes a cargo handling device 17. The cargo handling device 17 is located between a pair of left and right reach legs 14 at the front portion of the vehicle body 13. The cargo handling device 17 includes a pair of left and right outer masts 18 and an inner mast 19 that moves up and down with respect to the outer masts 18. Further, the cargo handling device 17 includes a lift cylinder 20 connected to the inner mast 19. The inner mast 19 moves up and down by supplying and discharging hydraulic oil to the lift cylinder 20. A lift bracket 21 that can be raised and lowered with respect to the inner mast 19 is provided. The lift bracket 21 is provided with a pair of left and right forks 22. The pair of left and right forks 22 can be tilted back and forth by a tilt cylinder (not shown).

As shown in FIG. 3, a backrest 23 is attached to the lift bracket 21 so as to be located above the fork 22. The backrest 23 supports the rear surface of the load when the load (not shown) is placed on the fork 22. The length of the backrest 23 in the left-right direction is set to a length that is smaller than the width of the vehicle body 13 but protrudes to the outside of the outer mast 18.

The vehicle body 13 is provided with a reach cylinder 24 connected to the cargo handling device 17 (see FIG. 2). The rod of the reach cylinder 24 moves back and forth in the front-rear direction due to the supply and discharge of hydraulic oil. By operating the reach cylinder 24, the cargo handling device 17 moves back and forth along the reach leg 14. The forklift 11 includes a traveling motor 25 for driving the rear wheels 16 and a steering motor (not shown) for steering the rear wheels 16.

As shown in FIG. 1, the forklift 11 has a radio unit 30 as a vehicle communication unit, a vehicle control unit 31, an image signal processing unit 32, and a camera 33. The wireless unit 30 is for performing wireless communication with the remote control device 12 described below. The vehicle control unit 31 is connected to each part of the forklift 11, controls each part connected to the vehicle control unit 31, and is connected to the radio unit 30. The image signal processing unit 32 signals the camera image captured by the camera 33 so as to be communicable.

The camera 33 photographs the tip of the fork 22 and the front of the fork 22, which are necessary for remote control of the forklift 11. As shown in FIGS. 3 and 4, the camera 33 is attached so that the lens faces forward at the center of the upper part of the backrest 23 which is an elevating part. Since the camera 33 is attached so as to face forward at the center of the upper part of the backrest 23, the tip of the fork 22 and the front of the fork 22 can be photographed by the camera 33. The camera 33 is a camera including a large number of CCD image sensors or C-MOS image sensors. Although not shown, the forklift 11 is provided with a camera 33, a plurality of traveling cameras for photographing the front and the rear for traveling, and a plurality of sensors (obstacle sensor, distance measuring sensor, etc.). ing.

Next, the remote control device 12 will be described. The remote control device 12 includes a wireless unit 34 as a communication unit, a control unit 35, an operation unit 36, a display unit 37, and an image signal processing unit 38. The wireless unit 34 can wirelessly communicate with the wireless unit 30 of the forklift 11. The control unit 35 is connected to the radio unit 34, the operation unit 36, the display unit 37, and the image signal processing unit 38.

The control unit 35 has a function of commanding the operation output of the operation unit 36 to the forklift 11. Although not shown, the control unit 35 has a CPU for executing various programs and a memory for storing various programs, data, and the like. The operation unit 36 includes a plurality of operation levers (not shown) and operation buttons (not shown). The operation output of the operation unit 36 is transmitted to the forklift 11 via the control unit 35. The operator can remotely control the forklift 11 by operating a plurality of operation levers and operation buttons.

The display unit 37 corresponds to a display monitor, for example, displays a camera image received through the wireless unit 34, and displays various information necessary for remote control. The image signal processing unit 38 processes the image recognition of the camera image taken by the camera 33, and also edits and processes the rearrangement of the camera images displayed on the display unit 37. The display unit 37 of the present embodiment includes a touch panel 39 capable of touch operation on the display screen (see FIG. 5). The touch panel 39 can specify an arbitrary position in the displayed camera image by a touch operation. Although it is assumed that the remote control device 12 has the display unit 37, the display monitor may be provided separately from the remote control device 12.

Then, when the operation unit 36 of the remote control device 12 is operated by the operator, the control unit 35 transmits the operation output to the forklift 11 via the radio unit 34. In the forklift 11, the operation output from the remote control device 12 is received by the radio unit 30, and the vehicle control unit 31 controls each part of the forklift 11 based on a command from the remote control device 12. Specifically, the traveling drive system equipment (travel motor 25, steering motor, etc.) and cargo handling equipment (lift cylinder 20, reach cylinder 24, tilt cylinder, etc.) are driven based on the command of the vehicle control unit 31. .. In this way, the remote control device 12 can remotely control the traveling of the forklift 11 and the cargo handling by the cargo handling device 17.

By the way, in the present embodiment, the camera image taken by the camera 33 is displayed on the display unit 37, but the operator can specify the target in the camera image displayed on the display unit 37. Specifically, the target is specified by the touch operation of the touch panel 39 by the operator. Therefore, the touch panel 39 corresponds to a target designation unit that specifies the target displayed in the camera image. The target is a destination point or an object for remote control required by the operator in cargo handling or traveling, and is included in the camera image displayed on the display unit 37 when the target is specified.

When the target is specified by the touch operation of the operator, the control unit 35 has a function of acquiring the world coordinates of the target. That is, the control unit 35 corresponds to the world coordinate acquisition unit that acquires the world coordinates of the target specified by the touch panel 39. Further, the control unit 35 corresponds to a guideline drawing unit that draws a guideline between the world coordinates of the forklift 11 and the world coordinates of the target on the camera image in the camera image. The guideline in the camera image is a drawing line superimposed on the camera screen so that at least the direction of the target seen from the forklift 11 can be intuitively grasped.

Here, the world coordinates will be described. The world coordinates have coordinate values given by the world coordinate system (Xw, Yw, Zw) which is a three-dimensional coordinate system. On the other hand, the screen coordinates have coordinate values given by the screen coordinate system (Xs, Ys) which is a two-dimensional coordinate system. A perspective projection conversion method is known as a method for converting world coordinates to screen coordinates. In the perspective projection conversion method, first, the field coordinates are converted to the viewpoint coordinates of the viewpoint coordinate system (Xe, Ye, Ze), and then the viewport conversion of the viewpoint coordinates is performed to screen coordinates. Is being converted to. The viewpoint coordinate system is a three-dimensional coordinate system based on the camera viewpoint. Therefore, the world coordinates indicating the position in space can be converted into screen coordinates indicating the two-dimensional position by the perspective projection conversion method and displayed on the screen.

In the present embodiment, the point C shown in FIG. 4 is a point indicating the world coordinates (Xwc, Ywc, Zwc) of the forklift 11. The world coordinates of the forklift 11 are acquired by the control unit 35. As shown in FIG. 5, in the camera image P1 taken by the camera 33 displayed on the display unit 37, a point C is drawn near the lower side of the center in the left-right direction of the screen on the display unit 37. The point C in the camera image P1 indicates the world coordinates (Xwc, Ywc, Zwc) and the screen coordinates (Xsc, Ysc) of the forklift 11.

Further, for example, when the pallet 40 displayed on the camera image P1 shown in FIG. 5 is targeted, pressing the portion where the pallet 40 is displayed on the touch panel 39 corresponds to the pressed portion on the camera image P1. T is displayed. The point T in the camera image P1 indicates the screen coordinates (Xst, Yst) and the world coordinates (Xwt, Ywt, Zwt) of the target palette 40 .

When the target is set, the control unit 35 acquires the world coordinates (Xwt, Ywt, Zwt) of the target and draws a guideline L of a straight line connecting the points C and T in the camera image P1 (FIG. 5). reference). When the forklift 11 is moved by remote control of the operator, the pallet 40, which is the target in the camera image P1, moves. Although the forklift 11 is actually moving, the control unit 35 performs a process of moving the point T relative to the point C, so that the value of the world coordinates of the point C is invariant, and the forklift 11 is moved. Along with this, the value of the world coordinates of the point T changes. Specifically, the control unit 35 grasps the movement amount of the forklift 11 based on the information from the vehicle control unit 31 of the forklift 11. The control unit 35 changes the world coordinates of the point T according to the grasped movement amount of the forklift 11. The control unit 35 constitutes a world coordinate changing unit that changes the world coordinates of the point T following the movement of the forklift 11. The control unit 35 draws the guideline L even if the forklift 11 moves. When the camera image P1 includes a point T, the guideline is a straight line connecting the point C to the target point T. Therefore, the guideline L makes it easy for the operator to intuitively grasp the position and direction of the target.

By the way, for example, as shown in FIG. 6, it is conceivable that the target pallet 40 deviates from the camera image P2 due to the remote control of the forklift 11. In FIG. 6, the palette 40 deviating from the camera image P2 is shown by a chain double-dashed line. Since the control unit 35 performs a process of connecting the world coordinates of the point C and the world coordinates of the point T, the guideline L is drawn within a drawable range. In FIG. 6, the point T deviates from the camera image P2, but the guideline L is the world coordinates of the point C (Xwc, Ywc, Zwc) and the world coordinates of the target point T (Xwt, Ywt, Zwt). It is drawn in the range of the camera image P2 between them. Therefore, the guidelines make it easier for the operator to at least intuitively understand the direction of the target.

Next, the remote control of the forklift 11 by the remote control system 10 of the forklift 11 according to the present embodiment will be described. When the operator remotely controls the forklift 11, the camera image P2 is displayed on the display unit 37, and when the operator operates the operation unit 36, the forklift 11 is remotely controlled according to the operator's operation.

As shown in FIG. 5, when the operator sets a target from the camera image P1 displayed on the display unit 37, a point T of the set target is generated in the camera image P1. The control unit 35 acquires the world coordinates by acquiring the distance from the camera 33 to the point T and then converting the screen coordinates of the point T into coordinates. By acquiring the world coordinates of the target point T, the guideline L connecting the point C which is the world coordinate of the forklift 11 and the point T which is the world coordinate of the target is drawn on the camera image P1.

When the forklift 11 is moved by remote control of the operator, the position of the target point T moves in the camera image P1, but the world coordinates of the point T are changed by the control unit 35 following the movement of the forklift 11, and the guideline L Is drawn in the camera image P1 when the forklift 11 is remotely controlled.

As shown in FIG. 6, even if the target pallet 40 deviates from the camera image P2 by remote control of the forklift 11, the guideline L continues to be displayed in the camera image P2. In this case, the guideline L indicates the direction of the target point T. As shown in FIG. 5, when the target is reflected in the camera image P1 again due to the traveling of the forklift 11, the guideline L connecting the point C of the forklift 11 and the point T of the target is displayed.

The remote control system 10 of the forklift 11 according to the present embodiment has the following effects.
(1) Between the point T (Xwt, Ywt, Zwt) as the world coordinate of the target specified by the touch panel 39 in the camera images P1 and P2 and the point C which is the world coordinate (Xwc, Ywc, Zwc) of the forklift 11. Guideline L is drawn on. Even if the target deviates from the camera image P2 due to the traveling of the forklift 11, the world coordinates of the point T are changed by the control unit 35 following the movement of the forklift 11, and the guideline L in the camera image P2 deviates from the camera image P2. Indicates the direction in which the target is present. Therefore, since the operator who remotely controls the forklift 11 can recognize the direction in which the target exists, the target will not be lost even if the target deviates from the camera image P2.

(2) The touch panel 39 whose target designation unit can be touch-operated on the display screen. Therefore, the operator can easily specify the target displayed on the camera image P1 by touching the touch panel 39.

(3) The guideline L is drawn by a straight line in the camera images P1 and P2. Therefore, the operator who remotely controls the forklift 11 can at least intuitively recognize the direction in which the target exists even if the target deviates from the camera image P2. Further, the guideline L connecting the point C (Xwc, Ywc, Zwc) as the world coordinates of the forklift 11 and the point T (Xwt, Ywt, Zwt) as the world coordinates of the target is the position and direction of the target with respect to the operator. Is easy to understand intuitively.

(Modification example)
Next, the guideline relating to the modified example will be described. As shown in FIG. 7, the guideline La of this modification is not a guideline connecting the point C of the forklift 11 and the point T of the target (see FIGS. 5 and 6), but the guideline La from the point C to the target in the camera image P3. It is an arrow line indicating the direction of the point T. The end on the target side in the arrow line is the end of the arrow, and the end on the forklift side is the end without the arrow.

The guideline La of this modification makes it easy for the operator to intuitively grasp the direction of the target. Further, since it is not necessary to change the shape of the guideline La regardless of whether or not the target deviates from the camera image P3, it is possible to simplify the control of drawing the guideline.

The present invention is not limited to the above-described embodiment (including modifications), and various modifications can be made within the scope of the gist of the invention. For example, the present invention may be modified as follows.

○ In the above embodiment, the guideline drawn on the camera image is a straight line, but this is not the case. For example, it may be a combination of curved straight lines or a curved line that bypasses an obstacle.
○ In the above embodiment, the guideline is continuously drawn during the remote control of the operator, but this is not the case, and the guideline may not be drawn depending on the timing. For example, the guideline may not be drawn when the target deviates from the camera image, and the guideline may be drawn when the target deviates from the camera image. In this case, the guideline does not interfere with the operator's perception of the object to be captured in the camera image when the target does not deviate from the camera image. Further, it may be a guideline that is displayed blinking or a semi-transparent guideline in which the transmittance is set.
○ In the above embodiment, the camera image taken in front of the forklift is displayed on the display monitor, but the present invention is not limited to this. For example, a pseudo bird's-eye view camera image (pseudo bird's-eye view camera image) is displayed on a display monitor by combining a plurality of camera images taken from the front, rear, and both sides, and the world coordinates and target of an industrial vehicle are displayed in the pseudo bird's-eye view camera image. You may draw a guideline with the world coordinates of.
○ In the above embodiment, an example of the touch panel has been described as the target designation unit, but the target designation unit is not limited to the touch panel. The target specification part may be a mouse or a keyboard, and the world coordinates of the target that is likely to be the target are registered in advance, the registered target is recognized on the camera screen, and the guideline is drawn. May be good.
○ In the above embodiment, a single target is set in the camera image, but this is not the case. For example, a plurality of targets may be set as a guideline for connecting the world coordinates of the industrial vehicle and the world coordinates of the plurality of targets.
○ In the above embodiment, an example of a forklift as an industrial vehicle has been described, but the purpose is not limited to a forklift, and the industrial vehicle may be a construction vehicle in addition to a cargo handling vehicle such as a forklift.

10 Forklift remote control system 11 Forklift 12 Remote control device 13 Body 17 Cargo handling device 22 Fork 32 Image signal processing unit 33 Camera 35 Control unit (world coordinate acquisition unit, guideline drawing unit , world coordinate change unit )
36 Operation unit 37 Display unit (display monitor)
38 Image signal processing unit 39 Touch panel (target specification unit)
40 Pallet E Workplace R Operation Room W Loads P1, P2, P3 Camera image point C (forklift world coordinates / screen coordinates)
Point T (target world coordinates / screen coordinates)

Claims (3)

An industrial vehicle equipped with a camera
A remote control device for remotely controlling the industrial vehicle and
In a remote control system of an industrial vehicle equipped with a display monitor provided with a display screen capable of displaying a camera image taken by the camera.
A target specification unit that specifies the target displayed in the camera image,
A world coordinate acquisition unit that acquires the world coordinates of the target specified by the target specification unit, and a world coordinate acquisition unit.
A world coordinate changing unit that changes the world coordinates of the target according to the movement of the industrial vehicle,
Remote control of an industrial vehicle, which comprises a guideline drawing unit for drawing a guideline between the world coordinates of the industrial vehicle and the world coordinates of the target changed by the world coordinate changing unit on the camera image. system. The remote control system for an industrial vehicle according to claim 1, wherein the target designation unit is a touch panel capable of touch operation on the display screen. The remote control system according to claim 1 or 2, wherein the guideline is drawn by a straight line in the camera image.

Images